Jul 15 21:30:03.315429 ip-10-0-135-178 systemd[1]: Starting Kubernetes Kubelet... Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.694890 2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699739 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699750 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699754 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699757 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699761 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699764 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699767 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:30:03.726684 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699771 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699776 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699779 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699782 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699785 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699788 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699791 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699794 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699797 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699800 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699803 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699806 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699808 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699811 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699813 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699816 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699819 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699821 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699824 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699829 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:30:03.728405 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699832 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699835 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699837 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699840 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699843 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699845 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699848 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699851 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699854 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699857 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699859 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699862 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699864 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699867 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699869 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699872 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699874 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699877 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699880 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699884 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:30:03.729268 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699886 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699889 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699892 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699894 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699897 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699899 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699902 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699904 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699907 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699909 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699912 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699915 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699917 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699920 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699923 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699925 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699928 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699930 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699933 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:30:03.730997 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699935 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699938 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699940 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699943 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699945 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699948 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699950 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699952 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699955 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699957 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699960 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699962 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699964 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699967 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699970 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699974 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699979 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699982 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699986 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699989 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:30:03.731868 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699991 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699995 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.699997 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700001 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700004 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700006 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700009 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700011 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700014 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700017 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700019 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700022 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700026 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700736 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700743 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700746 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700750 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700753 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700755 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700759 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:30:03.732726 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700765 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700768 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700771 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700774 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700776 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700779 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700783 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700786 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700789 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700791 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700794 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700796 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700799 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700801 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700804 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700807 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700809 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700812 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700814 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700817 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:30:03.733594 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700820 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700823 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700825 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700828 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700830 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700832 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700835 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700837 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700839 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700844 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700847 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700850 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700855 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700858 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700861 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700863 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700866 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700869 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700872 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:30:03.734457 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700875 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700878 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700881 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700884 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700886 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700889 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700892 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700895 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700897 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700902 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700906 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700909 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700912 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700916 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700918 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700921 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700923 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700926 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700928 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:30:03.735321 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700931 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700933 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700936 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700938 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700941 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700943 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700947 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700950 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700952 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700954 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700957 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700959 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700962 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700964 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700971 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700973 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700976 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700978 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700980 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700983 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:30:03.736203 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700986 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700988 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700990 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700993 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700995 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.700998 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701000 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701004 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701007 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701009 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701012 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701014 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701017 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701019 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701395 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701400 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701403 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701406 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701410 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701413 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701416 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:30:03.737489 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701418 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701421 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701424 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701426 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701429 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701432 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701434 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701438 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701441 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701443 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701446 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701449 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701452 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701454 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701457 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701459 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701462 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701464 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701468 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701470 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:30:03.738358 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701473 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701475 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701491 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701495 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701498 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701502 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701504 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701507 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701510 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701514 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701517 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701520 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701523 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701525 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701529 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701531 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701534 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701536 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701540 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701542 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:30:03.739450 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701545 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701548 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701550 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701554 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701557 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701560 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701563 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701566 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701568 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701571 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701578 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701581 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701583 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701586 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701588 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701591 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701593 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701596 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701598 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:30:03.740557 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701601 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701603 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701607 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701609 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701612 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701615 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701617 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701619 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701624 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701627 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701630 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701633 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701636 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701639 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701641 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701644 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701646 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701648 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701651 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:30:03.741434 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701654 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701656 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701658 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701661 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701664 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701666 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701669 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701672 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701674 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701676 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701679 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701681 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701684 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.701686 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701746 2572 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701754 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701761 2572 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701767 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701773 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701780 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701790 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 21:30:03.743159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701796 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701799 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701802 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701805 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701808 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701814 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701817 2572 flags.go:64] FLAG: --cgroup-root="" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701820 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701823 2572 flags.go:64] FLAG: --client-ca-file="" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701826 2572 flags.go:64] FLAG: --cloud-provider="external" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701829 2572 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701833 2572 flags.go:64] FLAG: --cluster-domain="" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701836 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701839 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701842 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701846 2572 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701850 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701853 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701856 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701860 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701863 2572 flags.go:64] FLAG: --contention-profiling="false" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701866 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701868 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701872 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 21:30:03.744029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701875 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701878 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701883 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701886 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701888 2572 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701891 2572 flags.go:64] FLAG: --enable-server="true" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701895 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701899 2572 flags.go:64] FLAG: --event-burst="100" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701902 2572 flags.go:64] FLAG: --event-qps="50" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701905 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701908 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701911 2572 flags.go:64] FLAG: --eviction-hard="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701914 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701917 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701921 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701924 2572 flags.go:64] FLAG: --eviction-soft="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701926 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701929 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701932 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701935 2572 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701938 2572 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701940 2572 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701944 2572 flags.go:64] FLAG: --feature-gates="" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701949 2572 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701952 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 21:30:03.746203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701955 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701958 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701961 2572 flags.go:64] FLAG: --healthz-port="10248" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701964 2572 flags.go:64] FLAG: --help="false" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701967 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-135-178.ec2.internal" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701970 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701973 2572 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701976 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701979 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701983 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701986 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701989 2572 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701991 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701994 2572 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.701998 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702001 2572 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702003 2572 flags.go:64] FLAG: --kube-reserved="" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702006 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702009 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702012 2572 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702015 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702017 2572 flags.go:64] FLAG: --lock-file="" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702021 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702024 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 21:30:03.747662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702027 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702033 2572 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702035 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702038 2572 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702041 2572 flags.go:64] FLAG: --logging-format="text" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702044 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702048 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702051 2572 flags.go:64] FLAG: --manifest-url="" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702054 2572 flags.go:64] FLAG: --manifest-url-header="" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702059 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702062 2572 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702066 2572 flags.go:64] FLAG: --max-pods="110" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702069 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702072 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702074 2572 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702077 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702080 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702085 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702087 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702099 2572 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702102 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702105 2572 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702108 2572 flags.go:64] FLAG: --pod-cidr="" Jul 15 21:30:03.749194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702111 2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702116 2572 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702119 2572 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702122 2572 flags.go:64] FLAG: --pods-per-core="0" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702125 2572 flags.go:64] FLAG: --port="10250" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702128 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702131 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01a2e77cff66102cf" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702134 2572 flags.go:64] FLAG: --qos-reserved="" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702137 2572 flags.go:64] FLAG: --read-only-port="10255" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702140 2572 flags.go:64] FLAG: --register-node="true" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702143 2572 flags.go:64] FLAG: --register-with-taints="" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702147 2572 flags.go:64] FLAG: --registry-burst="10" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702149 2572 flags.go:64] FLAG: --registry-qps="5" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702152 2572 flags.go:64] FLAG: --reserved-cpus="" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702155 2572 flags.go:64] FLAG: --reserved-memory="" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702174 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702180 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702183 2572 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702186 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702189 2572 flags.go:64] FLAG: --runonce="false" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702192 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702195 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702198 2572 flags.go:64] FLAG: --seccomp-default="false" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702201 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702204 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702207 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 21:30:03.750474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702210 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702214 2572 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702217 2572 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702220 2572 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702223 2572 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702226 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702229 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702232 2572 flags.go:64] FLAG: --system-cgroups="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702235 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702240 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702243 2572 flags.go:64] FLAG: --tls-cert-file="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702246 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702249 2572 flags.go:64] FLAG: --tls-min-version="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702252 2572 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702255 2572 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702258 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702261 2572 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702264 2572 flags.go:64] FLAG: --v="2" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702268 2572 flags.go:64] FLAG: --version="false" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702272 2572 flags.go:64] FLAG: --vmodule="" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702276 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.702280 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702409 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702413 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:30:03.751565 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702416 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702419 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702422 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702425 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702428 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702431 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702434 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702436 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702439 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702443 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702446 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702448 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702451 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702453 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702456 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702458 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702461 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702463 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702465 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702468 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:30:03.752649 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702471 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702473 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702475 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702478 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702481 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702483 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702486 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702488 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702491 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702494 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702497 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702499 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702502 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702504 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702506 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702509 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702511 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702514 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702516 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702518 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:30:03.753510 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702522 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702525 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702528 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702530 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702533 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702535 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702538 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702540 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702543 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702545 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702548 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702550 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702553 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702555 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702558 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702560 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702562 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702565 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702569 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:30:03.755051 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702573 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702576 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702580 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702582 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702586 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702589 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702592 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702595 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702598 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702600 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702603 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702605 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702608 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702610 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702614 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702617 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702620 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702623 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702625 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702628 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:30:03.755934 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702630 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702634 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702637 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702640 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702643 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702646 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702648 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702651 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702654 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702657 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702659 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702662 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702665 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702667 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702671 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702673 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702676 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:30:03.756830 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.702678 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.703124 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.710176 2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.710193 2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.710212 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.710218 2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710277 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710283 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710286 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710290 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710293 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710296 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710299 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710302 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710304 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:30:03.757717 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710307 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710311 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710314 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710317 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710321 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710323 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710326 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710328 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710331 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710333 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710336 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710339 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710341 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710344 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710346 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710349 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710351 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710353 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710356 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710358 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:30:03.758809 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710362 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710364 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710367 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710369 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710371 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710374 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710377 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710379 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710383 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710385 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710388 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710391 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710393 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710396 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710398 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710401 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710403 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710406 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710408 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710410 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:30:03.759683 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710413 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710415 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710418 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710422 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710425 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710427 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710429 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710432 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710435 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710437 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710440 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710442 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710444 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710447 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710450 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710452 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710455 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710457 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710460 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710462 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:30:03.760978 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710465 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710467 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710470 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710473 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710477 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710479 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710482 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710485 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710488 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710491 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710493 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710496 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710499 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710501 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710505 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710510 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710513 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710515 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710518 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:30:03.826987 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710520 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:30:03.810731 ip-10-0-135-178 systemd[1]: Started Kubernetes Kubelet. Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710523 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710525 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710528 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710530 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710533 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710535 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710538 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710540 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710543 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710545 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.710550 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710677 2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710682 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710685 2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 21:30:03.829065 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710688 2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710691 2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710693 2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710696 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710699 2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710701 2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710704 2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710707 2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710710 2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710712 2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710714 2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710717 2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710719 2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710722 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710726 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710728 2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710730 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710733 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710735 2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710738 2572 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 21:30:03.830085 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710740 2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710743 2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710745 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710748 2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710750 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710754 2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710756 2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710759 2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710761 2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710763 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710770 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710772 2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710774 2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710777 2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710779 2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710782 2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710784 2572 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710786 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710789 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710791 2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 21:30:03.831584 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710793 2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710796 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710799 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710802 2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710806 2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710809 2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710813 2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710815 2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710818 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710820 2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710833 2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710836 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710839 2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710841 2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710844 2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710846 2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710850 2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710853 2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710857 2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 21:30:03.832841 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710861 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710865 2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710869 2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710874 2572 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710880 2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710885 2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710890 2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710894 2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710898 2572 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710902 2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710906 2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710910 2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710913 2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710917 2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710920 2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710924 2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710928 2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710931 2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710935 2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 21:30:03.834472 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710942 2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710946 2572 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710950 2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710954 2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710957 2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710961 2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710965 2572 feature_gate.go:328] unrecognized feature gate: Example Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710968 2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710972 2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710976 2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710981 2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710986 2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710990 2572 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710995 2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.710999 2572 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.711003 2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.711009 2572 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 21:30:03.835438 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:03.711012 2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.711019 2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.711149 2572 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.713620 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.714967 2572 server.go:1019] "Starting client certificate rotation" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.715074 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.715114 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.716997 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.718450 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.778849 2572 log.go:25] "Validated CRI v1 runtime API" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.781869 2572 log.go:25] "Validated CRI v1 image API" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.782655 2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 21:30:03.836227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.784743 2572 fs.go:135] Filesystem UUIDs: map[17c15189-8ec1-46e5-92ff-d8056e59fe8b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9390c3f1-eaa9-4d78-816c-18d8d9b7b431:/dev/nvme0n1p4] Jul 15 21:30:03.836596 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.784766 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.790485 2572 manager.go:216] Machine: {Timestamp:2026-07-15 21:30:03.789245014 +0000 UTC m=+0.369842053 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198336 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214461d0249b3637942c2224199ccc SystemUUID:ec214461-d024-9b36-3794-2c2224199ccc BootID:1339af41-9e5e-4503-a89e-d46718a89c01 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f0:96:f7:a9:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f0:96:f7:a9:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:43:82:29:a0:68 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.790588 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.790668 2572 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.792329 2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.792362 2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-178.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.793087 2572 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.793099 2572 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 21:30:03.836660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.793109 2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.793695 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.795260 2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.797789 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798638 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798767 2572 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798807 2572 kubelet.go:486] "Attempting to sync node with API server" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798831 2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798849 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798860 2572 kubelet.go:398] "Adding apiserver pod source" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.798872 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.799262 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.799326 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.802552 2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805014 2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805048 2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805221 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805235 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805244 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805251 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805261 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805269 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805660 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805669 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805679 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805689 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805716 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.805732 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.806270 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.806289 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.809573 2572 server.go:1268] "Started kubelet" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.809647 2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.809774 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.810188 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.811598 2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.812341 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.812489 2572 server.go:310] "Adding debug handlers to kubelet server" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.812870 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.812903 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 21:30:03.836965 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.816375 2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.815512 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e297b99067 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.809550439 +0000 UTC m=+0.390147463,LastTimestamp:2026-07-15 21:30:03.809550439 +0000 UTC m=+0.390147463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819141 2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819176 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.819228 2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-178.ec2.internal\" not found" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819625 2572 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819649 2572 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819653 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819804 2572 reconstruct.go:97] "Volume reconstruction finished" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819811 2572 reconciler.go:29] "Reconciler: start to sync state" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819865 2572 factory.go:55] Registering systemd factory Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.819880 2572 factory.go:223] Registration of the systemd container factory successfully Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.820057 2572 factory.go:153] Registering CRI-O factory Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.820068 2572 factory.go:223] Registration of the crio container factory successfully Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.820116 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.820137 2572 factory.go:103] Registering Raw factory Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.820150 2572 manager.go:1196] Started watching for new ooms in manager Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.821084 2572 manager.go:349] Starting recovery of all containers Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.821248 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.821272 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.826297 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2j92n" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.829896 2572 manager.go:354] Recovery completed Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.835686 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.837455 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.837481 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.837491 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.838029 2572 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.838041 2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.838044 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:03.838323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.838056 2572 state_mem.go:36] "Initialized new in-memory state store" Jul 15 21:30:03.839492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.839092 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:03.839492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.839113 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:03.839492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.839124 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:03.839492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.839272 2572 policy_none.go:49] "None policy: Start" Jul 15 21:30:03.839492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.839288 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 21:30:03.839492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.839300 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 21:30:03.840437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.840308 2572 policy_none.go:47] "Start" Jul 15 21:30:03.898572 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.849586 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.898572 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.854576 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.898572 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.860232 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.898572 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.862740 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:03.839100948 +0000 UTC m=+0.419697967,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.898932 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.877947 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:03.839119318 +0000 UTC m=+0.419716337,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.898932 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.886532 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963f432\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:03.839128382 +0000 UTC m=+0.419725400,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:03.898932 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.893849 2572 manager.go:337] "Starting Device Plugin manager" Jul 15 21:30:03.898932 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.893888 2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 21:30:03.898932 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.893899 2572 server.go:87] "Starting device plugin registration server" Jul 15 21:30:03.901431 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.901343 2572 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 21:30:03.901431 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.901357 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 21:30:03.901817 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.901508 2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 21:30:03.902566 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.902429 2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 21:30:03.902566 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:03.902438 2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 21:30:03.902954 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.902771 2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 21:30:03.902954 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.902818 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-178.ec2.internal\" not found" Jul 15 21:30:03.904308 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:03.904033 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29ce8a62e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.896522286 +0000 UTC m=+0.477119298,LastTimestamp:2026-07-15 21:30:03.896522286 +0000 UTC m=+0.477119298,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.012208 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.002342 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.012208 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.003196 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.012208 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.003234 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.012208 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.003251 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.012208 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.003288 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.014247 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.014003 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:04.003213979 +0000 UTC m=+0.583810999,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.022688 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.022709 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:04.003242261 +0000 UTC m=+0.583839268,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.030059 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.030096 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963f432\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:04.003258303 +0000 UTC m=+0.583855322,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.047696 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.055958 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.055991 2572 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.056009 2572 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.056042 2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 21:30:04.124853 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.072125 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.156290 2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal"] Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.156392 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.158042 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.158075 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.158088 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.159488 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.159623 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.159650 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.160339 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.160350 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.160368 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.160376 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.160380 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.160390 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.161394 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.161419 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.162109 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.162127 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.162140 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.166784 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:04.158055257 +0000 UTC m=+0.738652281,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.226093 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.178712 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:04.158081781 +0000 UTC m=+0.738678802,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.189165 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963f432\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:04.158102501 +0000 UTC m=+0.738699523,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.198894 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.199229 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:04.160353443 +0000 UTC m=+0.740950465,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.203187 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.207845 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:04.160364197 +0000 UTC m=+0.740961219,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.217311 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:04.160373999 +0000 UTC m=+0.740971021,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.227916 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.222399 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e193e431239fbdd2a5f918e7173fd13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal\" (UID: \"2e193e431239fbdd2a5f918e7173fd13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.222422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e193e431239fbdd2a5f918e7173fd13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal\" (UID: \"2e193e431239fbdd2a5f918e7173fd13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.222441 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102fbd48ff7ff0cfcd397c73fd966f70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-178.ec2.internal\" (UID: \"102fbd48ff7ff0cfcd397c73fd966f70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.222868 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.223724 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.223752 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.223765 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.223795 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.231912 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.228393 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:04.160380933 +0000 UTC m=+0.740977954,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321523 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.267700 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.321523 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.267757 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963f432\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:04.160384515 +0000 UTC m=+0.740981537,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321523 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.271198 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963f432\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:04.160396324 +0000 UTC m=+0.740993332,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321523 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.278216 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:04.162118661 +0000 UTC m=+0.742715683,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321523 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.281689 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:04.162134925 +0000 UTC m=+0.742731950,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321994 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.289670 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963f432\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963f432 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837494322 +0000 UTC m=+0.418091340,LastTimestamp:2026-07-15 21:30:04.162144748 +0000 UTC m=+0.742741771,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321994 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.303765 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e299638cda\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e299638cda default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837467866 +0000 UTC m=+0.418064889,LastTimestamp:2026-07-15 21:30:04.223737162 +0000 UTC m=+0.804334184,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.321994 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.314189 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-178.ec2.internal.18c293e29963d309\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-178.ec2.internal.18c293e29963d309 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-178.ec2.internal,UID:ip-10-0-135-178.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-178.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:03.837485833 +0000 UTC m=+0.418082851,LastTimestamp:2026-07-15 21:30:04.223757168 +0000 UTC m=+0.804354252,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:04.323409 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.322635 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e193e431239fbdd2a5f918e7173fd13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal\" (UID: \"2e193e431239fbdd2a5f918e7173fd13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.323409 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.322659 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e193e431239fbdd2a5f918e7173fd13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal\" (UID: \"2e193e431239fbdd2a5f918e7173fd13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.323409 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.322677 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102fbd48ff7ff0cfcd397c73fd966f70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-178.ec2.internal\" (UID: \"102fbd48ff7ff0cfcd397c73fd966f70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.323409 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.322723 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/102fbd48ff7ff0cfcd397c73fd966f70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-178.ec2.internal\" (UID: \"102fbd48ff7ff0cfcd397c73fd966f70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.323409 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.322751 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e193e431239fbdd2a5f918e7173fd13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal\" (UID: \"2e193e431239fbdd2a5f918e7173fd13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.323409 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.322751 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e193e431239fbdd2a5f918e7173fd13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal\" (UID: \"2e193e431239fbdd2a5f918e7173fd13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.455574 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.455538 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms" Jul 15 21:30:04.501839 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.501777 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.507082 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.506013 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.671437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.671389 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.672516 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.672483 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.672663 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.672528 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.672663 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.672546 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:04.672663 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.672577 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.692943 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.692913 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:04.821330 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.821267 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:04.904910 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:04.904880 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 21:30:04.940905 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.940879 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:04.941818 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.941800 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:04.941898 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.941834 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:04.941898 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:04.941847 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:05.137107 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.137079 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 21:30:05.225336 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:05.225303 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e193e431239fbdd2a5f918e7173fd13.slice/crio-5d0db30f97a9d81b785c25f04c229bc396560134e43bd17afda02716f456516e WatchSource:0}: Error finding container 5d0db30f97a9d81b785c25f04c229bc396560134e43bd17afda02716f456516e: Status 404 returned error can't find the container with id 5d0db30f97a9d81b785c25f04c229bc396560134e43bd17afda02716f456516e Jul 15 21:30:05.228432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.228418 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:30:05.237460 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.237374 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e2ec4eefda openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:05.228625882 +0000 UTC m=+1.809222894,LastTimestamp:2026-07-15 21:30:05.228625882 +0000 UTC m=+1.809222894,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:05.265492 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.265471 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s" Jul 15 21:30:05.298117 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.298091 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 21:30:05.300950 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:05.300925 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102fbd48ff7ff0cfcd397c73fd966f70.slice/crio-3f3d78e2ba04be5db4077363841e9cc087ec46f524c32573a8c8a983aa0f5a93 WatchSource:0}: Error finding container 3f3d78e2ba04be5db4077363841e9cc087ec46f524c32573a8c8a983aa0f5a93: Status 404 returned error can't find the container with id 3f3d78e2ba04be5db4077363841e9cc087ec46f524c32573a8c8a983aa0f5a93 Jul 15 21:30:05.310993 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.310917 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-178.ec2.internal.18c293e2f0b6fa85 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-178.ec2.internal,UID:102fbd48ff7ff0cfcd397c73fd966f70,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0595b1d6de8e08ddb61ce581075357331193db6206d341c15cf213514ef0f693\",Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:05.302553221 +0000 UTC m=+1.883150229,LastTimestamp:2026-07-15 21:30:05.302553221 +0000 UTC m=+1.883150229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:05.493862 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.493786 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:05.494901 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.494882 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:05.495009 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.494920 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:05.495009 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.494936 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:05.495009 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.494975 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:05.515213 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.515169 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:05.550533 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:05.550507 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 15 21:30:05.824936 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:05.824827 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:06.062093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:06.062030 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" event={"ID":"102fbd48ff7ff0cfcd397c73fd966f70","Type":"ContainerStarted","Data":"3f3d78e2ba04be5db4077363841e9cc087ec46f524c32573a8c8a983aa0f5a93"} Jul 15 21:30:06.063980 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:06.063947 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" event={"ID":"2e193e431239fbdd2a5f918e7173fd13","Type":"ContainerStarted","Data":"5d0db30f97a9d81b785c25f04c229bc396560134e43bd17afda02716f456516e"} Jul 15 21:30:06.821343 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:06.821304 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:06.876599 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:06.876561 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s" Jul 15 21:30:07.115857 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.115779 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:07.116885 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.116849 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:07.117016 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.116892 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:07.117016 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.116908 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:07.117016 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.116944 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:07.134402 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.134375 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:07.177741 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.177706 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:07.179256 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.179222 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:07.179399 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.179268 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:07.179399 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.179285 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:07.249517 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.249481 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 21:30:07.383080 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.382722 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-178.ec2.internal.18c293e36c23a167 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-178.ec2.internal,UID:102fbd48ff7ff0cfcd397c73fd966f70,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0595b1d6de8e08ddb61ce581075357331193db6206d341c15cf213514ef0f693\" in 2.07s (2.07s including waiting). Image size: 303035524 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.373271399 +0000 UTC m=+3.953868407,LastTimestamp:2026-07-15 21:30:07.373271399 +0000 UTC m=+3.953868407,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:07.391676 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.391595 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e36c405d42 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\" in 2.146s (2.146s including waiting). Image size: 264138005 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.375154498 +0000 UTC m=+3.955751506,LastTimestamp:2026-07-15 21:30:07.375154498 +0000 UTC m=+3.955751506,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:07.450603 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.450528 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-178.ec2.internal.18c293e370284f04 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-178.ec2.internal,UID:102fbd48ff7ff0cfcd397c73fd966f70,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Created container: haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.440686852 +0000 UTC m=+4.021283882,LastTimestamp:2026-07-15 21:30:07.440686852 +0000 UTC m=+4.021283882,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:07.458247 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.458158 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-135-178.ec2.internal.18c293e370a3e379 kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-135-178.ec2.internal,UID:102fbd48ff7ff0cfcd397c73fd966f70,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Started container haproxy,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.448785785 +0000 UTC m=+4.029382795,LastTimestamp:2026-07-15 21:30:07.448785785 +0000 UTC m=+4.029382795,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:07.583371 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.583337 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 21:30:07.583371 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.583286 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e378184946 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.573854534 +0000 UTC m=+4.154451555,LastTimestamp:2026-07-15 21:30:07.573854534 +0000 UTC m=+4.154451555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:07.592070 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.592001 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e3789c8191 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.582519697 +0000 UTC m=+4.163116723,LastTimestamp:2026-07-15 21:30:07.582519697 +0000 UTC m=+4.163116723,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:07.821403 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:07.821339 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:07.937112 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:07.937067 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 21:30:08.069263 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.069220 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" event={"ID":"102fbd48ff7ff0cfcd397c73fd966f70","Type":"ContainerStarted","Data":"09d229e0bfc488b33ea38189bbcafa314fa402d80a877144bbbac89478e5e802"} Jul 15 21:30:08.069438 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.069245 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:08.070361 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.070340 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:08.070468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.070408 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:08.070468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.070422 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:08.070628 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:08.070611 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:08.071196 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.071172 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" event={"ID":"2e193e431239fbdd2a5f918e7173fd13","Type":"ContainerStarted","Data":"da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a"} Jul 15 21:30:08.071279 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.071263 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:08.072005 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.071957 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:08.072005 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.071985 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:08.072005 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.071996 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:08.072156 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:08.072148 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:08.459598 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:08.459519 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 15 21:30:08.821778 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:08.821695 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:09.072686 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.072603 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:09.073060 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.072603 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:09.073569 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.073547 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:09.073680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.073582 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:09.073680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.073598 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:09.073680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.073604 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:09.073680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.073623 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:09.073680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.073636 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:09.073901 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:09.073866 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:09.073969 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:09.073952 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:09.823792 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:09.823757 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:10.085749 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:10.085669 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s" Jul 15 21:30:10.335027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:10.334993 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:10.336036 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:10.335973 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:10.336036 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:10.336011 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:10.336036 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:10.336026 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:10.336265 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:10.336063 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:10.354144 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:10.354115 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:10.821776 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:10.821691 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:11.205876 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:11.205768 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:11.206712 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:11.206695 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:11.206789 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:11.206726 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:11.206789 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:11.206735 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:11.241049 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:11.241023 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 21:30:11.605347 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:11.605276 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 21:30:11.821019 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:11.820995 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:12.642550 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:12.642514 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 21:30:12.821114 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:12.821073 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:12.985822 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:12.985751 2572 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Jul 15 21:30:13.821894 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:13.821864 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:13.903277 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:13.903237 2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-178.ec2.internal\" not found" Jul 15 21:30:14.820517 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:14.820486 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:15.825916 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:15.825886 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:16.494999 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:16.494966 2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="7s" Jul 15 21:30:16.754353 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:16.754270 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:16.755236 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:16.755216 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:16.755350 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:16.755251 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:16.755350 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:16.755265 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:16.755350 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:16.755295 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:16.775226 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:16.775203 2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-135-178.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:16.822457 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:16.822443 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:17.820336 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:17.820311 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:18.820184 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:18.820150 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-178.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 21:30:19.088447 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.088378 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal_2e193e431239fbdd2a5f918e7173fd13/setup/0.log" Jul 15 21:30:19.088447 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.088412 2572 generic.go:356] "Generic (PLEG): container finished" podID="2e193e431239fbdd2a5f918e7173fd13" containerID="da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a" exitCode=1 Jul 15 21:30:19.088447 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.088439 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" event={"ID":"2e193e431239fbdd2a5f918e7173fd13","Type":"ContainerDied","Data":"da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a"} Jul 15 21:30:19.088860 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.088560 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:19.089372 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.089357 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:19.089419 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.089385 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:19.089419 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.089394 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:19.089566 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:19.089555 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:19.100449 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:19.100380 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e626a55719 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Container image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\" already present on machine,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:19.092268825 +0000 UTC m=+15.672865842,LastTimestamp:2026-07-15 21:30:19.092268825 +0000 UTC m=+15.672865842,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:19.168247 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:19.168160 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e378184946\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e378184946 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Created container: setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.573854534 +0000 UTC m=+4.154451555,LastTimestamp:2026-07-15 21:30:19.160633188 +0000 UTC m=+15.741230195,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:19.178373 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:19.178302 2572 event.go:359] "Server rejected event (will not retry!)" err="events \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e3789c8191\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal.18c293e3789c8191 openshift-machine-config-operator 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal,UID:2e193e431239fbdd2a5f918e7173fd13,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Started container setup,Source:EventSource{Component:kubelet,Host:ip-10-0-135-178.ec2.internal,},FirstTimestamp:2026-07-15 21:30:07.582519697 +0000 UTC m=+4.163116723,LastTimestamp:2026-07-15 21:30:19.16832802 +0000 UTC m=+15.748925051,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-178.ec2.internal,}" Jul 15 21:30:19.311105 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.311063 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:19.312028 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.312007 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:19.312126 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.312046 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:19.312126 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.312059 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:19.537785 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.537654 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2j92n" Jul 15 21:30:19.714513 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.714483 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 21:30:19.827680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.827623 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:19.844968 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.844950 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:19.904909 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.904891 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:19.999572 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:19.999554 2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:30:20.091235 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.091181 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal_2e193e431239fbdd2a5f918e7173fd13/setup/0.log" Jul 15 21:30:20.091235 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.091221 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" event={"ID":"2e193e431239fbdd2a5f918e7173fd13","Type":"ContainerStarted","Data":"e8aea419cc897c080cf86399bd675ebf76d8bf57b9e15e6edc1ca7c422dbaead"} Jul 15 21:30:20.091578 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.091322 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:20.092278 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.092263 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:20.092358 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.092291 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:20.092358 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.092301 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:20.092487 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:20.092476 2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:20.174263 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.174241 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.174263 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:20.174265 2572 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.212002 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.211985 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.229257 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.229242 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.288472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.288457 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.449583 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.449536 2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:30:20.541880 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.541850 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 21:25:19 +0000 UTC" deadline="2028-01-17 08:57:06.238993389 +0000 UTC" Jul 15 21:30:20.541880 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.541877 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13211h26m45.697118784s" Jul 15 21:30:20.549869 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.549856 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.549909 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:20.549874 2572 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.818433 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.818372 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.833071 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.833054 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:20.892759 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:20.892741 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:21.165612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:21.165549 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:21.165612 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:21.165576 2572 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:21.599616 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:21.599543 2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:30:21.811460 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:21.811424 2572 apiserver.go:52] "Watching apiserver" Jul 15 21:30:21.813561 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:21.813546 2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 21:30:21.813658 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:21.813641 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=[] Jul 15 21:30:21.820558 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:21.820548 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 21:30:23.258708 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.258674 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:23.274356 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.274336 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:23.330313 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.330294 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:23.501715 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:23.501686 2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-178.ec2.internal\" not found" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:23.609077 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.609020 2572 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:23.609077 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:23.609041 2572 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-135-178.ec2.internal" not found Jul 15 21:30:23.776154 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.776128 2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 21:30:23.777447 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.777431 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientMemory" Jul 15 21:30:23.777530 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.777458 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 21:30:23.777530 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.777469 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeHasSufficientPID" Jul 15 21:30:23.777530 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.777494 2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:23.785720 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.785703 2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-178.ec2.internal" Jul 15 21:30:23.785779 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:23.785728 2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-178.ec2.internal\": node \"ip-10-0-135-178.ec2.internal\" not found" Jul 15 21:30:23.809346 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.809327 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-p45dp"] Jul 15 21:30:23.812325 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.812313 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-tfzw6"] Jul 15 21:30:23.812442 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.812428 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.814814 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.814798 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.815274 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.815249 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 21:30:23.815384 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.815281 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-96tz9\"" Jul 15 21:30:23.815647 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.815633 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 21:30:23.818804 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.818511 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 21:30:23.818804 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.818602 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 21:30:23.818804 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.818698 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 21:30:23.818804 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.818698 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 21:30:23.818804 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.818743 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 21:30:23.819288 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.819272 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" Jul 15 21:30:23.819397 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.819379 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dmhfn\"" Jul 15 21:30:23.829940 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.829924 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 21:30:23.842316 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.842296 2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 21:30:23.870612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.870574 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vk5dk" Jul 15 21:30:23.879646 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.879630 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vk5dk" Jul 15 21:30:23.888930 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.888915 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.888991 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.888938 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w2xx\" (UniqueName: \"kubernetes.io/projected/bae5c93e-952f-4779-9030-8d96e59ced89-kube-api-access-9w2xx\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.888991 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.888953 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7639f6d8-7349-44cd-8ce9-25027172896c-hosts-file\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.888991 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.888966 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7639f6d8-7349-44cd-8ce9-25027172896c-tmp-dir\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.888991 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.888981 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgsc\" (UniqueName: \"kubernetes.io/projected/7639f6d8-7349-44cd-8ce9-25027172896c-kube-api-access-ntgsc\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.889187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.889006 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-system-cni-dir\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.889187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.889028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-os-release\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.889187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.889050 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.889187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.889097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-cnibin\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.889187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.889135 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-cni-binary-copy\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.889187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.889181 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.900710 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.900694 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:30:23.900785 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.900743 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal"] Jul 15 21:30:23.900785 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.900773 2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" Jul 15 21:30:23.903796 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.903780 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-wrbz2"] Jul 15 21:30:23.905995 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.905979 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:23.907630 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.907613 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-9l9v7"] Jul 15 21:30:23.908370 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.908341 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 21:30:23.908577 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.908563 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 21:30:23.908657 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.908580 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jlxr2\"" Jul 15 21:30:23.908657 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.908635 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:30:23.909976 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.909961 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:23.910047 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:23.910027 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:23.922462 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.922442 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-7qtb7"] Jul 15 21:30:23.924685 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.924673 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:23.924734 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:23.924718 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:23.927702 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.927689 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bssvn"] Jul 15 21:30:23.929822 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.929809 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:23.932159 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.932143 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 21:30:23.932284 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.932267 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 21:30:23.932341 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.932312 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lhxlv\"" Jul 15 21:30:23.932456 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.932442 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 21:30:23.944115 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.944100 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-89q4j"] Jul 15 21:30:23.946330 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.946310 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89q4j" Jul 15 21:30:23.948817 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.948802 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 21:30:23.948900 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.948814 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5zldt\"" Jul 15 21:30:23.990122 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990102 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-cnibin\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990126 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-cni-binary-copy\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990141 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w2xx\" (UniqueName: \"kubernetes.io/projected/bae5c93e-952f-4779-9030-8d96e59ced89-kube-api-access-9w2xx\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990174 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7639f6d8-7349-44cd-8ce9-25027172896c-tmp-dir\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.990333 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990202 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:23.990333 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990244 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed84b377-e8fa-4089-8a99-79436b6fa9b4-serviceca\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:23.990333 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990288 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92bbe301-b932-42f7-939c-c1f424fd41ff-iptables-alerter-script\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:23.990333 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990310 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92bbe301-b932-42f7-939c-c1f424fd41ff-host-slash\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990322 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-cnibin\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990375 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990394 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7639f6d8-7349-44cd-8ce9-25027172896c-hosts-file\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990428 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgsc\" (UniqueName: \"kubernetes.io/projected/7639f6d8-7349-44cd-8ce9-25027172896c-kube-api-access-ntgsc\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.990501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7639f6d8-7349-44cd-8ce9-25027172896c-tmp-dir\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990506 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed84b377-e8fa-4089-8a99-79436b6fa9b4-host\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7639f6d8-7349-44cd-8ce9-25027172896c-hosts-file\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-system-cni-dir\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990559 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-os-release\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990569 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990576 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990619 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-system-cni-dir\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990652 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bae5c93e-952f-4779-9030-8d96e59ced89-os-release\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990654 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khv5h\" (UniqueName: \"kubernetes.io/projected/ed84b377-e8fa-4089-8a99-79436b6fa9b4-kube-api-access-khv5h\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990693 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-cni-binary-copy\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990701 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgqrz\" (UniqueName: \"kubernetes.io/projected/92bbe301-b932-42f7-939c-c1f424fd41ff-kube-api-access-qgqrz\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:23.990808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990746 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxp9\" (UniqueName: \"kubernetes.io/projected/16cfb664-2ac4-4852-8f0b-85b1422267e6-kube-api-access-pmxp9\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:23.991134 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990833 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.991134 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.990910 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bae5c93e-952f-4779-9030-8d96e59ced89-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:23.996988 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.996975 2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 21:30:23.997100 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:23.997087 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal"] Jul 15 21:30:24.005090 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.005072 2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 21:30:24.008631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.008610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w2xx\" (UniqueName: \"kubernetes.io/projected/bae5c93e-952f-4779-9030-8d96e59ced89-kube-api-access-9w2xx\") pod \"multus-additional-cni-plugins-tfzw6\" (UID: \"bae5c93e-952f-4779-9030-8d96e59ced89\") " pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:24.009661 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.009646 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgsc\" (UniqueName: \"kubernetes.io/projected/7639f6d8-7349-44cd-8ce9-25027172896c-kube-api-access-ntgsc\") pod \"node-resolver-p45dp\" (UID: \"7639f6d8-7349-44cd-8ce9-25027172896c\") " pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:24.010042 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.010026 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-j6tlt"] Jul 15 21:30:24.013707 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.013685 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm"] Jul 15 21:30:24.015227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.015193 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.016474 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.016458 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.017628 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.017615 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 21:30:24.017840 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.017826 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m5jtd\"" Jul 15 21:30:24.017877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.017859 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 21:30:24.019313 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.019298 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 21:30:24.019371 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.019335 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 21:30:24.019501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.019486 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 21:30:24.019547 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.019529 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p2t5g\"" Jul 15 21:30:24.046199 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.046151 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-178.ec2.internal" podStartSLOduration=1.046140119 podStartE2EDuration="1.046140119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:24.046055865 +0000 UTC m=+20.626652903" watchObservedRunningTime="2026-07-15 21:30:24.046140119 +0000 UTC m=+20.626737146" Jul 15 21:30:24.091751 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.091721 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-multus-certs\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.091854 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.091757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-registration-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.091854 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.091781 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-etc-selinux\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.091854 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.091810 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khv5h\" (UniqueName: \"kubernetes.io/projected/ed84b377-e8fa-4089-8a99-79436b6fa9b4-kube-api-access-khv5h\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.091854 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.091833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgqrz\" (UniqueName: \"kubernetes.io/projected/92bbe301-b932-42f7-939c-c1f424fd41ff-kube-api-access-qgqrz\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.092035 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.091862 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-sys-fs\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.092089 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092030 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-system-cni-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092089 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092056 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-cnibin\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-device-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.092152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092122 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b27b4e0-f870-47ec-a705-c370b082ce9a-cni-binary-copy\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092135 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-cni-bin\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092292 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092156 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-cni-multus\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092292 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092197 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed84b377-e8fa-4089-8a99-79436b6fa9b4-serviceca\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.092292 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092220 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92bbe301-b932-42f7-939c-c1f424fd41ff-iptables-alerter-script\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.092292 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092243 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92bbe301-b932-42f7-939c-c1f424fd41ff-host-slash\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092290 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/92bbe301-b932-42f7-939c-c1f424fd41ff-host-slash\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092299 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-socket-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092332 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-daemon-config\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092357 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7a08fd4-44f6-4065-b984-e0a1a5554cdd-konnectivity-ca\") pod \"konnectivity-agent-j6tlt\" (UID: \"e7a08fd4-44f6-4065-b984-e0a1a5554cdd\") " pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092384 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed84b377-e8fa-4089-8a99-79436b6fa9b4-host\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092406 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-netns\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-hostroot\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092452 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed84b377-e8fa-4089-8a99-79436b6fa9b4-host\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.092466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092454 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwgk\" (UniqueName: \"kubernetes.io/projected/3b27b4e0-f870-47ec-a705-c370b082ce9a-kube-api-access-rmwgk\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092484 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7a08fd4-44f6-4065-b984-e0a1a5554cdd-agent-certs\") pod \"konnectivity-agent-j6tlt\" (UID: \"e7a08fd4-44f6-4065-b984-e0a1a5554cdd\") " pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092517 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxp9\" (UniqueName: \"kubernetes.io/projected/16cfb664-2ac4-4852-8f0b-85b1422267e6-kube-api-access-pmxp9\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092559 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-cni-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092584 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-os-release\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092614 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092643 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxg8\" (UniqueName: \"kubernetes.io/projected/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-kube-api-access-wzxg8\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-k8s-cni-cncf-io\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-kubelet\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092791 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-conf-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-etc-kubernetes\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:24.092877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-socket-dir-parent\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.093266 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.092883 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ed84b377-e8fa-4089-8a99-79436b6fa9b4-serviceca\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.093266 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.092909 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:24.093266 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.092964 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:24.592947303 +0000 UTC m=+21.173544323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:24.093266 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.093110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/92bbe301-b932-42f7-939c-c1f424fd41ff-iptables-alerter-script\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.103487 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.103473 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:24.103487 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.103489 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:24.103592 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.103497 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:24.103592 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.103539 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:24.603525709 +0000 UTC m=+21.184122719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:24.104706 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.104688 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khv5h\" (UniqueName: \"kubernetes.io/projected/ed84b377-e8fa-4089-8a99-79436b6fa9b4-kube-api-access-khv5h\") pod \"node-ca-bssvn\" (UID: \"ed84b377-e8fa-4089-8a99-79436b6fa9b4\") " pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.106505 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.106476 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-lnzxm"] Jul 15 21:30:24.108538 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.108518 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgqrz\" (UniqueName: \"kubernetes.io/projected/92bbe301-b932-42f7-939c-c1f424fd41ff-kube-api-access-qgqrz\") pod \"iptables-alerter-wrbz2\" (UID: \"92bbe301-b932-42f7-939c-c1f424fd41ff\") " pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.109286 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.109269 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxp9\" (UniqueName: \"kubernetes.io/projected/16cfb664-2ac4-4852-8f0b-85b1422267e6-kube-api-access-pmxp9\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:24.110356 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.110339 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-4vkfh"] Jul 15 21:30:24.110455 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.110440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.113011 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.112985 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.113099 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.113084 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 21:30:24.114334 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.114315 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 21:30:24.114429 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.114371 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 21:30:24.114429 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.114389 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 21:30:24.114429 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.114418 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 21:30:24.114594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.114422 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 21:30:24.114594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.114389 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5jt85\"" Jul 15 21:30:24.115466 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.115440 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-svqdc\"" Jul 15 21:30:24.115545 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.115475 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:30:24.115545 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.115492 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 21:30:24.122557 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.122517 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-p45dp" Jul 15 21:30:24.126024 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.126008 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-tg8tn"] Jul 15 21:30:24.126064 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.126047 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" Jul 15 21:30:24.129151 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.129137 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.129211 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.129198 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:24.131672 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.131653 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7639f6d8_7349_44cd_8ce9_25027172896c.slice/crio-3ddac0fdcd6d9586fac26b9e8ec7bb571ad093745a3ef01175fc63d86c54edf8 WatchSource:0}: Error finding container 3ddac0fdcd6d9586fac26b9e8ec7bb571ad093745a3ef01175fc63d86c54edf8: Status 404 returned error can't find the container with id 3ddac0fdcd6d9586fac26b9e8ec7bb571ad093745a3ef01175fc63d86c54edf8 Jul 15 21:30:24.134188 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.134159 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae5c93e_952f_4779_9030_8d96e59ced89.slice/crio-848b0d81bc43c897ddf0df872bd885ed1bb236b76ff46b5629ffbc17e1f0f993 WatchSource:0}: Error finding container 848b0d81bc43c897ddf0df872bd885ed1bb236b76ff46b5629ffbc17e1f0f993: Status 404 returned error can't find the container with id 848b0d81bc43c897ddf0df872bd885ed1bb236b76ff46b5629ffbc17e1f0f993 Jul 15 21:30:24.193832 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7a08fd4-44f6-4065-b984-e0a1a5554cdd-agent-certs\") pod \"konnectivity-agent-j6tlt\" (UID: \"e7a08fd4-44f6-4065-b984-e0a1a5554cdd\") " pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.193930 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193845 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7lgc\" (UniqueName: \"kubernetes.io/projected/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-kube-api-access-k7lgc\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.193930 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193868 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1d9bff4-d3f4-47ea-9951-48073e0519a9-kubelet-config\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.193930 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-cni-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.194027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-k8s-cni-cncf-io\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193979 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-etc-kubernetes\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.193999 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-systemd-units\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194021 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-env-overrides\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194042 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovn-node-metrics-cert\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194072 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-socket-dir-parent\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194091 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-cni-bin\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194112 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysctl-conf\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194133 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-slash\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194159 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-sys-fs\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.194203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194198 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-system-cni-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194232 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-cni-bin\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194253 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-cni-multus\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194267 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-cni-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b27b4e0-f870-47ec-a705-c370b082ce9a-cni-binary-copy\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bstk4\" (UniqueName: \"kubernetes.io/projected/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-kube-api-access-bstk4\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194375 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-kubernetes\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysctl-d\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194414 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-tmp\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194438 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-socket-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194457 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-daemon-config\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovnkube-script-lib\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194504 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-systemd\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194519 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-var-lib-kubelet\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194535 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-log-socket\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194549 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-run\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194563 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-lib-modules\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194579 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-os-release\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1d9bff4-d3f4-47ea-9951-48073e0519a9-dbus\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194609 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxg8\" (UniqueName: \"kubernetes.io/projected/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-kube-api-access-wzxg8\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194626 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-kubelet\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-conf-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194657 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-kubelet\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194673 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-run-netns\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194688 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-node-log\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194701 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-tuned\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-multus-certs\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-registration-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194746 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-etc-selinux\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194762 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-systemd\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194776 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-host\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.194887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194786 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-socket-dir-parent\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwgk\" (UniqueName: \"kubernetes.io/projected/3b27b4e0-f870-47ec-a705-c370b082ce9a-kube-api-access-rmwgk\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194826 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-cnibin\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194871 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-ovn\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194891 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysconfig\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-device-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194937 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7a08fd4-44f6-4065-b984-e0a1a5554cdd-konnectivity-ca\") pod \"konnectivity-agent-j6tlt\" (UID: \"e7a08fd4-44f6-4065-b984-e0a1a5554cdd\") " pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194958 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-sys\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194979 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195000 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-etc-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195013 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195025 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-var-lib-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195043 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-k8s-cni-cncf-io\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195046 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195068 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-etc-kubernetes\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195068 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-cni-netd\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195100 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovnkube-config\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.194714 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3b27b4e0-f870-47ec-a705-c370b082ce9a-cni-binary-copy\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195209 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-conf-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195232 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-kubelet\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195250 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-cni-bin\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195258 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-cnibin\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195259 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-sys-fs\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195255 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-modprobe-d\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195292 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-registration-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195333 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-multus-certs\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195367 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-var-lib-cni-multus\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195369 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-system-cni-dir\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.195873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-device-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195555 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-etc-selinux\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195670 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-os-release\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195713 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-socket-dir\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195771 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-netns\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-hostroot\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.195833 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-host-run-netns\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.196030 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3b27b4e0-f870-47ec-a705-c370b082ce9a-multus-daemon-config\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.196294 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3b27b4e0-f870-47ec-a705-c370b082ce9a-hostroot\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.196364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.196320 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e7a08fd4-44f6-4065-b984-e0a1a5554cdd-agent-certs\") pod \"konnectivity-agent-j6tlt\" (UID: \"e7a08fd4-44f6-4065-b984-e0a1a5554cdd\") " pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.196599 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.196409 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e7a08fd4-44f6-4065-b984-e0a1a5554cdd-konnectivity-ca\") pod \"konnectivity-agent-j6tlt\" (UID: \"e7a08fd4-44f6-4065-b984-e0a1a5554cdd\") " pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.202501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.202480 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwgk\" (UniqueName: \"kubernetes.io/projected/3b27b4e0-f870-47ec-a705-c370b082ce9a-kube-api-access-rmwgk\") pod \"multus-89q4j\" (UID: \"3b27b4e0-f870-47ec-a705-c370b082ce9a\") " pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.203007 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.202988 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxg8\" (UniqueName: \"kubernetes.io/projected/dc3cada3-d7c0-40d5-a25f-f16f03fc6945-kube-api-access-wzxg8\") pod \"aws-ebs-csi-driver-node-rt8cm\" (UID: \"dc3cada3-d7c0-40d5-a25f-f16f03fc6945\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.215467 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.215454 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wrbz2" Jul 15 21:30:24.232917 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.232891 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92bbe301_b932_42f7_939c_c1f424fd41ff.slice/crio-0605c2d817e15f24d90907f03053aceb91318c812bf09f7945e38026a6790ee1 WatchSource:0}: Error finding container 0605c2d817e15f24d90907f03053aceb91318c812bf09f7945e38026a6790ee1: Status 404 returned error can't find the container with id 0605c2d817e15f24d90907f03053aceb91318c812bf09f7945e38026a6790ee1 Jul 15 21:30:24.237469 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.237050 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bssvn" Jul 15 21:30:24.245985 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.245965 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded84b377_e8fa_4089_8a99_79436b6fa9b4.slice/crio-f039386dfb1b73870ad124d02ced65a8c575e04492ac260b03e4cf585805e023 WatchSource:0}: Error finding container f039386dfb1b73870ad124d02ced65a8c575e04492ac260b03e4cf585805e023: Status 404 returned error can't find the container with id f039386dfb1b73870ad124d02ced65a8c575e04492ac260b03e4cf585805e023 Jul 15 21:30:24.253343 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.253327 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-89q4j" Jul 15 21:30:24.260669 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.260652 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b27b4e0_f870_47ec_a705_c370b082ce9a.slice/crio-ddbbdda368349aa04bf85e5329e7b80d94464b79cbb86b765272dad1a1baa932 WatchSource:0}: Error finding container ddbbdda368349aa04bf85e5329e7b80d94464b79cbb86b765272dad1a1baa932: Status 404 returned error can't find the container with id ddbbdda368349aa04bf85e5329e7b80d94464b79cbb86b765272dad1a1baa932 Jul 15 21:30:24.296501 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-systemd\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-var-lib-kubelet\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-log-socket\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296536 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-run\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296585 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-log-socket\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296594 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-lib-modules\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296605 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-var-lib-kubelet\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296598 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-systemd\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1d9bff4-d3f4-47ea-9951-48073e0519a9-dbus\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296658 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-run\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296678 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-kubelet\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296705 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-run-netns\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-node-log\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296748 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-run-netns\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-tuned\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296782 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-systemd\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-host\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296835 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-kubelet\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296848 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296862 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-node-log\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296866 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b1d9bff4-d3f4-47ea-9951-48073e0519a9-dbus\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-ovn\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.296941 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296900 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-host\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-systemd\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296911 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-ovn\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296913 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-lib-modules\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296926 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-run-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296945 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysconfig\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.296971 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-sys\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297005 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297018 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysconfig\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297030 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-etc-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297033 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-sys\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297056 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-var-lib-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297080 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297085 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-etc-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297124 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-var-lib-openvswitch\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.297590 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.297127 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297176 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-cni-netd\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297127 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.297210 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:24.797189887 +0000 UTC m=+21.377786904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297213 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-cni-netd\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297237 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovnkube-config\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297259 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-modprobe-d\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297285 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7lgc\" (UniqueName: \"kubernetes.io/projected/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-kube-api-access-k7lgc\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297307 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1d9bff4-d3f4-47ea-9951-48073e0519a9-kubelet-config\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297343 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-systemd-units\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297385 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-env-overrides\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297408 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovn-node-metrics-cert\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297417 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b1d9bff4-d3f4-47ea-9951-48073e0519a9-kubelet-config\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297421 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-systemd-units\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298125 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297441 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-cni-bin\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297459 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysctl-conf\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297385 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-modprobe-d\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297479 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-slash\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297509 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bstk4\" (UniqueName: \"kubernetes.io/projected/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-kube-api-access-bstk4\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297531 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-kubernetes\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297526 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-cni-bin\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysctl-d\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297578 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-tmp\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-slash\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297606 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297633 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysctl-conf\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297695 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-kubernetes\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297726 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-sysctl-d\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovnkube-script-lib\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297796 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovnkube-config\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298954 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297812 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298954 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.297800 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-env-overrides\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.298954 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.298322 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovnkube-script-lib\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.299464 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.299449 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-tmp\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.299521 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.299479 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-etc-tuned\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.299521 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.299511 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-ovn-node-metrics-cert\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.306992 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.306974 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7lgc\" (UniqueName: \"kubernetes.io/projected/393e2123-f0d2-4c27-ac7e-9fcfb66bd655-kube-api-access-k7lgc\") pod \"tuned-4vkfh\" (UID: \"393e2123-f0d2-4c27-ac7e-9fcfb66bd655\") " pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.307324 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.307307 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bstk4\" (UniqueName: \"kubernetes.io/projected/9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1-kube-api-access-bstk4\") pod \"ovnkube-node-lnzxm\" (UID: \"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.323145 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.323127 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:24.327949 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.327935 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" Jul 15 21:30:24.333628 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.333609 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a08fd4_44f6_4065_b984_e0a1a5554cdd.slice/crio-d52de7069c6e832aa2a421fe284cafc1bee3303098799f2608cd21e06328cc37 WatchSource:0}: Error finding container d52de7069c6e832aa2a421fe284cafc1bee3303098799f2608cd21e06328cc37: Status 404 returned error can't find the container with id d52de7069c6e832aa2a421fe284cafc1bee3303098799f2608cd21e06328cc37 Jul 15 21:30:24.336295 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.336275 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3cada3_d7c0_40d5_a25f_f16f03fc6945.slice/crio-fae82cf42a1b81eb0012987c5c6cefe86990d139ccf1fe5b74f6afc46aa8f6da WatchSource:0}: Error finding container fae82cf42a1b81eb0012987c5c6cefe86990d139ccf1fe5b74f6afc46aa8f6da: Status 404 returned error can't find the container with id fae82cf42a1b81eb0012987c5c6cefe86990d139ccf1fe5b74f6afc46aa8f6da Jul 15 21:30:24.420951 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.420872 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:24.425824 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.425810 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" Jul 15 21:30:24.430180 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.430144 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c893c1a_a131_4b5e_ba1d_74e25f8eb7d1.slice/crio-6ec1e1680a4c04a8bbb30e44438341cd0ec240bd7a7465ad12914cf2cf793aa2 WatchSource:0}: Error finding container 6ec1e1680a4c04a8bbb30e44438341cd0ec240bd7a7465ad12914cf2cf793aa2: Status 404 returned error can't find the container with id 6ec1e1680a4c04a8bbb30e44438341cd0ec240bd7a7465ad12914cf2cf793aa2 Jul 15 21:30:24.434119 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:24.434101 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393e2123_f0d2_4c27_ac7e_9fcfb66bd655.slice/crio-23ad05a941496ab4b020c60175f781ab123c218e8589d5c1db991acfa67cc9ac WatchSource:0}: Error finding container 23ad05a941496ab4b020c60175f781ab123c218e8589d5c1db991acfa67cc9ac: Status 404 returned error can't find the container with id 23ad05a941496ab4b020c60175f781ab123c218e8589d5c1db991acfa67cc9ac Jul 15 21:30:24.600110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.600082 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:24.600282 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.600205 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:24.600282 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.600260 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:25.600245506 +0000 UTC m=+22.180842528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:24.701575 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.701537 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:24.701750 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.701730 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:24.701750 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.701748 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:24.701852 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.701761 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:24.701852 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.701831 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:25.701802607 +0000 UTC m=+22.282399629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:24.802053 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.802018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:24.802263 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.802239 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:24.802325 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:24.802306 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:25.802288808 +0000 UTC m=+22.382885837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:24.882319 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.882254 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 21:25:23 +0000 UTC" deadline="2028-03-06 16:50:33.73646358 +0000 UTC" Jul 15 21:30:24.882319 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.882317 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14395h20m8.854150362s" Jul 15 21:30:24.977109 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:24.977035 2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 21:30:25.103761 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.103707 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal_2e193e431239fbdd2a5f918e7173fd13/setup/0.log" Jul 15 21:30:25.103912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.103762 2572 generic.go:356] "Generic (PLEG): container finished" podID="2e193e431239fbdd2a5f918e7173fd13" containerID="e8aea419cc897c080cf86399bd675ebf76d8bf57b9e15e6edc1ca7c422dbaead" exitCode=0 Jul 15 21:30:25.103912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.103868 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" event={"ID":"2e193e431239fbdd2a5f918e7173fd13","Type":"ContainerDied","Data":"e8aea419cc897c080cf86399bd675ebf76d8bf57b9e15e6edc1ca7c422dbaead"} Jul 15 21:30:25.104032 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.103975 2572 scope.go:117] "RemoveContainer" containerID="da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a" Jul 15 21:30:25.104496 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.104478 2572 scope.go:117] "RemoveContainer" containerID="da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a" Jul 15 21:30:25.116741 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.116709 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" event={"ID":"393e2123-f0d2-4c27-ac7e-9fcfb66bd655","Type":"ContainerStarted","Data":"23ad05a941496ab4b020c60175f781ab123c218e8589d5c1db991acfa67cc9ac"} Jul 15 21:30:25.121760 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.121734 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"6ec1e1680a4c04a8bbb30e44438341cd0ec240bd7a7465ad12914cf2cf793aa2"} Jul 15 21:30:25.127007 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.126979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" event={"ID":"dc3cada3-d7c0-40d5-a25f-f16f03fc6945","Type":"ContainerStarted","Data":"fae82cf42a1b81eb0012987c5c6cefe86990d139ccf1fe5b74f6afc46aa8f6da"} Jul 15 21:30:25.129403 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.129346 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6tlt" event={"ID":"e7a08fd4-44f6-4065-b984-e0a1a5554cdd","Type":"ContainerStarted","Data":"d52de7069c6e832aa2a421fe284cafc1bee3303098799f2608cd21e06328cc37"} Jul 15 21:30:25.133541 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.133514 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bssvn" event={"ID":"ed84b377-e8fa-4089-8a99-79436b6fa9b4","Type":"ContainerStarted","Data":"f039386dfb1b73870ad124d02ced65a8c575e04492ac260b03e4cf585805e023"} Jul 15 21:30:25.136099 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.136077 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wrbz2" event={"ID":"92bbe301-b932-42f7-939c-c1f424fd41ff","Type":"ContainerStarted","Data":"0605c2d817e15f24d90907f03053aceb91318c812bf09f7945e38026a6790ee1"} Jul 15 21:30:25.142410 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.142383 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89q4j" event={"ID":"3b27b4e0-f870-47ec-a705-c370b082ce9a","Type":"ContainerStarted","Data":"ddbbdda368349aa04bf85e5329e7b80d94464b79cbb86b765272dad1a1baa932"} Jul 15 21:30:25.145888 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.145851 2572 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal_openshift-machine-config-operator_2e193e431239fbdd2a5f918e7173fd13_0 in pod sandbox 5d0db30f97a9d81b785c25f04c229bc396560134e43bd17afda02716f456516e from index: no such id: 'da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a'" containerID="da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a" Jul 15 21:30:25.145978 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.145914 2572 kuberuntime_container.go:962] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal_openshift-machine-config-operator_2e193e431239fbdd2a5f918e7173fd13_0 in pod sandbox 5d0db30f97a9d81b785c25f04c229bc396560134e43bd17afda02716f456516e from index: no such id: 'da6a2c8e50b643ca93e7ecab39e9efa9ad44e7b55d1ba6460c468eb851c4d03a'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal_openshift-machine-config-operator(2e193e431239fbdd2a5f918e7173fd13)\"" logger="UnhandledError" Jul 15 21:30:25.146529 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.146505 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerStarted","Data":"848b0d81bc43c897ddf0df872bd885ed1bb236b76ff46b5629ffbc17e1f0f993"} Jul 15 21:30:25.148847 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.148823 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p45dp" event={"ID":"7639f6d8-7349-44cd-8ce9-25027172896c","Type":"ContainerStarted","Data":"3ddac0fdcd6d9586fac26b9e8ec7bb571ad093745a3ef01175fc63d86c54edf8"} Jul 15 21:30:25.609597 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.609516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:25.610019 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.609643 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:25.610019 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.609708 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:27.609686836 +0000 UTC m=+24.190283844 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:25.710840 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.710803 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:25.710999 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.710981 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:25.711068 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.711006 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:25.711068 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.711019 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:25.711183 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.711082 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:27.711063218 +0000 UTC m=+24.291660243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:25.811879 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.811845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:25.812057 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.811971 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:25.812057 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:25.812038 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:27.812018711 +0000 UTC m=+24.392615741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:25.882629 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.882538 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 21:25:23 +0000 UTC" deadline="2028-03-12 18:34:45.667972191 +0000 UTC" Jul 15 21:30:25.882629 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:25.882572 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14541h4m19.785404132s" Jul 15 21:30:26.059022 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:26.058993 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:26.059203 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:26.059111 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:26.059536 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:26.059514 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:26.059650 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:26.059630 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:26.059709 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:26.059694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:26.059811 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:26.059780 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:26.160277 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:26.160191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" event={"ID":"2e193e431239fbdd2a5f918e7173fd13","Type":"ContainerStarted","Data":"dc0466d60eae6b16b3275234b60caec8d1652cdf1b90ec846d6754bf83b25e3b"} Jul 15 21:30:26.175798 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:26.175751 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-178.ec2.internal" podStartSLOduration=3.175726001 podStartE2EDuration="3.175726001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:26.174431984 +0000 UTC m=+22.755029032" watchObservedRunningTime="2026-07-15 21:30:26.175726001 +0000 UTC m=+22.756323029" Jul 15 21:30:27.627123 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:27.627088 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:27.630969 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.630942 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:27.631101 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.631048 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:31.63102418 +0000 UTC m=+28.211621200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:27.729029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:27.728400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:27.729029 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.728578 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:27.729029 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.728598 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:27.729029 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.728610 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:27.729029 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.728671 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:31.728651002 +0000 UTC m=+28.309248017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:27.828994 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:27.828955 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:27.829155 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.829132 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:27.829251 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:27.829211 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:31.829192838 +0000 UTC m=+28.409789863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:28.057034 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:28.056714 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:28.057034 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:28.056814 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:28.057034 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:28.056833 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:28.057034 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:28.056953 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:28.057034 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:28.056717 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:28.057427 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:28.057043 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:30.056839 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:30.056807 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:30.057315 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:30.056807 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:30.057315 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:30.056947 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:30.057315 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:30.057039 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:30.057315 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:30.057093 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:30.057315 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:30.057183 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:31.664470 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:31.664429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:31.664898 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.664594 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:31.664898 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.664670 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:39.664649139 +0000 UTC m=+36.245246168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:31.765761 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:31.765238 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:31.765761 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.765461 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:31.765761 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.765480 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:31.765761 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.765494 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:31.765761 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.765547 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:39.765529494 +0000 UTC m=+36.346126503 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:31.866361 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:31.866217 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:31.866551 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.866411 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:31.866551 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:31.866473 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:39.866454658 +0000 UTC m=+36.447051664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:32.056536 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:32.056505 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:32.056693 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:32.056632 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:32.057093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:32.057072 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:32.057254 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:32.057201 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:32.057304 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:32.057267 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:32.057385 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:32.057358 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:34.057624 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:34.057387 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:34.058035 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:34.057443 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:34.058035 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:34.057723 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:34.058035 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:34.057481 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:34.058035 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:34.057802 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:34.058035 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:34.057871 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:35.910880 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.910854 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qdkxd"] Jul 15 21:30:35.913950 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.913923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:35.916449 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.916402 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-g2fkk\"" Jul 15 21:30:35.916590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.916563 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 21:30:35.917608 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.917512 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 21:30:35.917608 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.917535 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 21:30:35.917608 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.917574 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 21:30:35.917808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.917535 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 21:30:35.917808 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:35.917512 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 21:30:36.056476 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.056396 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:36.056476 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.056435 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:36.056749 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:36.056534 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:36.056749 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.056576 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:36.056749 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:36.056660 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:36.056850 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:36.056747 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:36.101150 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101120 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-textfile\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101150 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-metrics-client-ca\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101375 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101205 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-wtmp\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101375 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101228 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101375 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101250 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2g6q\" (UniqueName: \"kubernetes.io/projected/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-kube-api-access-m2g6q\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101375 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101365 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-sys\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101542 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101387 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-tls\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101542 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101407 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.101542 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.101430 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-root\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202521 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202493 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-root\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202521 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202522 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-textfile\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202545 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-metrics-client-ca\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202590 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-wtmp\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202606 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-root\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202624 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2g6q\" (UniqueName: \"kubernetes.io/projected/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-kube-api-access-m2g6q\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202776 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-sys\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202802 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-wtmp\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.202828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202820 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-tls\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.203220 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.202849 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.203220 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.203092 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-textfile\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.203220 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.203109 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-sys\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.203369 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.203302 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-metrics-client-ca\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.203369 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.203329 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.206233 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.206215 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-tls\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.206346 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.206230 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.218612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.218565 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2g6q\" (UniqueName: \"kubernetes.io/projected/e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a-kube-api-access-m2g6q\") pod \"node-exporter-qdkxd\" (UID: \"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a\") " pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:36.223223 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:36.223204 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qdkxd" Jul 15 21:30:38.057287 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:38.057216 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:38.057724 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:38.057216 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:38.057724 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:38.057341 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:38.057724 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:38.057230 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:38.057724 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:38.057416 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:38.057724 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:38.057515 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:39.727279 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:39.727242 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:39.727689 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.727390 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:39.727689 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.727469 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:55.727449457 +0000 UTC m=+52.308046479 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:39.828627 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:39.828596 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:39.828812 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.828784 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:39.828812 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.828805 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:39.828895 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.828815 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:39.828895 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.828867 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:55.828853519 +0000 UTC m=+52.409450529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:39.929180 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:39.929137 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:39.929371 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.929312 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:39.929432 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:39.929389 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:30:55.929369056 +0000 UTC m=+52.509966064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:40.056322 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:40.056225 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:40.056322 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:40.056259 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:40.056519 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:40.056341 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:40.056519 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:40.056416 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:40.056519 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:40.056440 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:40.056636 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:40.056538 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:42.057150 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:42.057119 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:42.057667 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:42.057119 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:42.057667 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:42.057230 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:42.057667 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:42.057257 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:42.057667 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:42.057375 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:42.057667 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:42.057466 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:42.207540 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:42.207509 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0fc3ebe_e8a0_4ef5_ade7_35d187cdd29a.slice/crio-6466c6f449039621e7d5fbe6eeb370db016c8534ac6a06f98779461f7ae54d9c WatchSource:0}: Error finding container 6466c6f449039621e7d5fbe6eeb370db016c8534ac6a06f98779461f7ae54d9c: Status 404 returned error can't find the container with id 6466c6f449039621e7d5fbe6eeb370db016c8534ac6a06f98779461f7ae54d9c Jul 15 21:30:43.197087 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.196706 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:30:43.197786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197450 2572 generic.go:356] "Generic (PLEG): container finished" podID="9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1" containerID="bc4a637a7e3b3424b11c22eef8029d30a0ffc46e7f1beea24b5265ff5f966767" exitCode=1 Jul 15 21:30:43.197786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197513 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"79cd9d4e91670092c3d93582103092f61796442d8dfc4c032d900feafec29a6c"} Jul 15 21:30:43.197786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197664 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"2409457f3ff3b96c2b9f386a8c02c16261f3bb732a902b236314c1c36b831201"} Jul 15 21:30:43.197786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197728 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"7cfcb417ad4f1f6567d1aab75e5e482c97799530f38dc26f48e7c97ccfe34053"} Jul 15 21:30:43.197988 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197814 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"2667d52f860f2e26a63d325f7e7c97e95669bff6b50ebe25812838bc346dfbb8"} Jul 15 21:30:43.197988 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197880 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerDied","Data":"bc4a637a7e3b3424b11c22eef8029d30a0ffc46e7f1beea24b5265ff5f966767"} Jul 15 21:30:43.197988 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.197913 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"3e78d4bcbcb01bcccb6188509e5db724b23642a67f183187117ad25175cbed35"} Jul 15 21:30:43.199076 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.199032 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" event={"ID":"dc3cada3-d7c0-40d5-a25f-f16f03fc6945","Type":"ContainerStarted","Data":"f44fac72c128b4cda93817127e43c52f451a8a867aecd2009655c161695bc53a"} Jul 15 21:30:43.200620 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.200598 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j6tlt" event={"ID":"e7a08fd4-44f6-4065-b984-e0a1a5554cdd","Type":"ContainerStarted","Data":"037fa6c15c3524dc858cc60f985505440c792207110d83524b098b8862ee95a8"} Jul 15 21:30:43.201978 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.201957 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bssvn" event={"ID":"ed84b377-e8fa-4089-8a99-79436b6fa9b4","Type":"ContainerStarted","Data":"a104ab62a7c52fccd6951b21f08c41bdc172f49fde891c93e2f6d73f47f2ae27"} Jul 15 21:30:43.203486 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.203460 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-89q4j" event={"ID":"3b27b4e0-f870-47ec-a705-c370b082ce9a","Type":"ContainerStarted","Data":"1bc01772ed0ad9ee55e7fe72f3c2f60832fae4e6be2c1716ad2c64b7bc1dab75"} Jul 15 21:30:43.205061 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.205039 2572 generic.go:356] "Generic (PLEG): container finished" podID="bae5c93e-952f-4779-9030-8d96e59ced89" containerID="19a732c22a089736218c807cee59f458c0a782d4b9e475695c618a3159c3062b" exitCode=0 Jul 15 21:30:43.205180 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.205106 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerDied","Data":"19a732c22a089736218c807cee59f458c0a782d4b9e475695c618a3159c3062b"} Jul 15 21:30:43.206726 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.206623 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-p45dp" event={"ID":"7639f6d8-7349-44cd-8ce9-25027172896c","Type":"ContainerStarted","Data":"41ffa89a5df87acf02ee032a056820c2504e5a2f133c19b2e672e03fcb23d32e"} Jul 15 21:30:43.208449 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.208429 2572 generic.go:356] "Generic (PLEG): container finished" podID="e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a" containerID="1cb073fccc905e6b09811dcf0f0c495e4f7ecf5ec5d4d6bc8b8df8e5577ab2b2" exitCode=0 Jul 15 21:30:43.208655 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.208512 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxd" event={"ID":"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a","Type":"ContainerDied","Data":"1cb073fccc905e6b09811dcf0f0c495e4f7ecf5ec5d4d6bc8b8df8e5577ab2b2"} Jul 15 21:30:43.208655 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.208552 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxd" event={"ID":"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a","Type":"ContainerStarted","Data":"6466c6f449039621e7d5fbe6eeb370db016c8534ac6a06f98779461f7ae54d9c"} Jul 15 21:30:43.210210 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.210186 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" event={"ID":"393e2123-f0d2-4c27-ac7e-9fcfb66bd655","Type":"ContainerStarted","Data":"f74bd73949074c3f971820b1414daf4800998bd5c425f7ac60cc8039af255b87"} Jul 15 21:30:43.214778 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.214742 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j6tlt" podStartSLOduration=2.38712651 podStartE2EDuration="20.214726651s" totalImagesPullingTime="17.827600141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:43.213916163 +0000 UTC m=+39.794513192" watchObservedRunningTime="2026-07-15 21:30:43.214726651 +0000 UTC m=+39.795323680" Jul 15 21:30:43.243341 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.243301 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-p45dp" podStartSLOduration=6.696562377 podStartE2EDuration="20.243287138s" totalImagesPullingTime="13.546724761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:43.243058779 +0000 UTC m=+39.823655807" watchObservedRunningTime="2026-07-15 21:30:43.243287138 +0000 UTC m=+39.823884166" Jul 15 21:30:43.294467 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.294416 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bssvn" podStartSLOduration=2.378558816 podStartE2EDuration="20.294395699s" totalImagesPullingTime="17.915836883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:43.278719452 +0000 UTC m=+39.859316479" watchObservedRunningTime="2026-07-15 21:30:43.294395699 +0000 UTC m=+39.874992731" Jul 15 21:30:43.295027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.295000 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-89q4j" podStartSLOduration=2.350439176 podStartE2EDuration="20.294987855s" totalImagesPullingTime="17.944548679s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:43.294007785 +0000 UTC m=+39.874604815" watchObservedRunningTime="2026-07-15 21:30:43.294987855 +0000 UTC m=+39.875584887" Jul 15 21:30:43.313004 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.312955 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4vkfh" podStartSLOduration=1.5438953789999998 podStartE2EDuration="19.312935551s" totalImagesPullingTime="17.769040172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:43.312193202 +0000 UTC m=+39.892790257" watchObservedRunningTime="2026-07-15 21:30:43.312935551 +0000 UTC m=+39.893532578" Jul 15 21:30:43.492473 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.492434 2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 21:30:43.748889 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.748859 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:43.802319 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.802287 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:43.934373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.934248 2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T21:30:43.49245592Z","UUID":"82f66120-6de9-4eef-947f-e4d9f0b5b507","Handler":null,"Name":"","Endpoint":""} Jul 15 21:30:43.936661 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.936634 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 21:30:43.936760 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:43.936670 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 21:30:44.057351 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.057319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:44.057466 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:44.057412 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:44.057525 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.057511 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:44.057582 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.057537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:44.057651 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:44.057629 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:44.057751 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:44.057727 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:44.214284 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.214250 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxd" event={"ID":"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a","Type":"ContainerStarted","Data":"c2100b3c03a831ff6a20d1536b098feda0700a41c9e08303e8e9fcceb1d0aed8"} Jul 15 21:30:44.214950 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.214307 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qdkxd" event={"ID":"e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a","Type":"ContainerStarted","Data":"e321c01ef39da7f70c39fe448644793b6cef1e6791055bd533c51b3addaa17d6"} Jul 15 21:30:44.215884 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.215849 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" event={"ID":"dc3cada3-d7c0-40d5-a25f-f16f03fc6945","Type":"ContainerStarted","Data":"77574f497f971b010b56b810297de28746027458cda728a1be537987318bf2a6"} Jul 15 21:30:44.219744 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.219708 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wrbz2" event={"ID":"92bbe301-b932-42f7-939c-c1f424fd41ff","Type":"ContainerStarted","Data":"37d473d6527949a51c7ab0031b97e42b600a00cdc4ba1aab59e38d905f40806f"} Jul 15 21:30:44.233350 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.233312 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qdkxd" podStartSLOduration=8.36460537 podStartE2EDuration="9.233299553s" totalImagesPullingTime="796.796744ms" totalInitContainerRuntime="71.897439ms" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:44.233047859 +0000 UTC m=+40.813644886" watchObservedRunningTime="2026-07-15 21:30:44.233299553 +0000 UTC m=+40.813896581" Jul 15 21:30:44.247417 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.247379 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wrbz2" podStartSLOduration=7.801911294 podStartE2EDuration="21.247367596s" totalImagesPullingTime="13.445456302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:44.247186025 +0000 UTC m=+40.827783061" watchObservedRunningTime="2026-07-15 21:30:44.247367596 +0000 UTC m=+40.827964623" Jul 15 21:30:44.323396 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.323373 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:44.324137 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.324120 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:44.341890 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.341867 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:44.402576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:44.402548 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:45.224173 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.224133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" event={"ID":"dc3cada3-d7c0-40d5-a25f-f16f03fc6945","Type":"ContainerStarted","Data":"9bb02087ea6376a9eb8400878adde9ef01402949715506936b1fb408f0d944d7"} Jul 15 21:30:45.227471 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.227442 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:30:45.227870 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.227843 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"c2254d17d9a44eac29a2c6da6ec1cce2e9d80ea64a6928e16e455cf37de21ab8"} Jul 15 21:30:45.228116 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.228087 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:45.228594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.228579 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j6tlt" Jul 15 21:30:45.241553 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.241514 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rt8cm" podStartSLOduration=2.288918578 podStartE2EDuration="22.24150308s" totalImagesPullingTime="19.952584502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:45.240949455 +0000 UTC m=+41.821546482" watchObservedRunningTime="2026-07-15 21:30:45.24150308 +0000 UTC m=+41.822100107" Jul 15 21:30:45.296705 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.296674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:45.561308 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.561237 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:45.934103 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:45.933858 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:46.057110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:46.057077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:46.057292 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:46.057077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:46.057292 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:46.057220 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:46.057292 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:46.057081 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:46.057442 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:46.057286 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:46.057442 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:46.057359 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:47.391645 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:47.391600 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:48.056691 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:48.056657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:48.056836 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:48.056662 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:48.056836 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:48.056797 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:48.056954 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:48.056873 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:48.056954 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:48.056903 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:48.057042 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:48.057004 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:48.449479 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:48.449453 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:48.551203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:48.551159 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:49.238561 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.238531 2572 generic.go:356] "Generic (PLEG): container finished" podID="bae5c93e-952f-4779-9030-8d96e59ced89" containerID="739dea43cecbda60e76e441620c26487b702c8899a325b0e76f1acd3d8c839d1" exitCode=0 Jul 15 21:30:49.238712 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.238617 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerDied","Data":"739dea43cecbda60e76e441620c26487b702c8899a325b0e76f1acd3d8c839d1"} Jul 15 21:30:49.244186 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.244155 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:30:49.244566 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.244544 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"63b3afc54b579acf469db1f1fb28aa025274d74f043a3e2054506ccaf47de6fe"} Jul 15 21:30:49.244748 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.244724 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:49.244841 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.244779 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:49.244932 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.244919 2572 scope.go:117] "RemoveContainer" containerID="bc4a637a7e3b3424b11c22eef8029d30a0ffc46e7f1beea24b5265ff5f966767" Jul 15 21:30:49.260778 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:49.260629 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:50.057222 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.057189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:50.057608 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.057189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:50.057608 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:50.057306 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:50.057608 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.057189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:50.057608 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:50.057387 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:50.057608 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:50.057454 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:50.251979 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.251946 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:30:50.252388 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.252354 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" event={"ID":"9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1","Type":"ContainerStarted","Data":"5263c19fa507b9911d7c5e8a432ed30f6b2656cff006df4600ccc9271c772343"} Jul 15 21:30:50.252557 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.252538 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:50.269779 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.269751 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:30:50.282107 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.282047 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" podStartSLOduration=8.44951579 podStartE2EDuration="26.28203018s" totalImagesPullingTime="17.83251439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:30:50.281306385 +0000 UTC m=+46.861903412" watchObservedRunningTime="2026-07-15 21:30:50.28203018 +0000 UTC m=+46.862627208" Jul 15 21:30:50.606651 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.606620 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:50.695649 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.695628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qtb7"] Jul 15 21:30:50.695762 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.695751 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:50.695858 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:50.695839 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:50.697976 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.697955 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9l9v7"] Jul 15 21:30:50.698086 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.698069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:50.698204 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:50.698186 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:50.711256 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.711236 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tg8tn"] Jul 15 21:30:50.711340 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:50.711317 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:50.711403 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:50.711388 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:51.065911 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:51.065880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:51.256064 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:51.255992 2572 generic.go:356] "Generic (PLEG): container finished" podID="bae5c93e-952f-4779-9030-8d96e59ced89" containerID="4fa8089883484ee02abd2e7bc0feab5baf01721dc88c26d5979bbb377c099cc7" exitCode=0 Jul 15 21:30:51.256227 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:51.256082 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerDied","Data":"4fa8089883484ee02abd2e7bc0feab5baf01721dc88c26d5979bbb377c099cc7"} Jul 15 21:30:52.056528 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:52.056497 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:52.056528 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:52.056511 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:52.056724 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:52.056521 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:52.056724 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:52.056616 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:52.056724 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:52.056664 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:52.056870 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:52.056769 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:52.260137 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:52.260111 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerStarted","Data":"027aea239f69b569105beabb2ea461b6582abbec0cc28e12c4d9c9108fad26c7"} Jul 15 21:30:52.683109 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:52.682945 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:53.264836 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:53.264804 2572 generic.go:356] "Generic (PLEG): container finished" podID="bae5c93e-952f-4779-9030-8d96e59ced89" containerID="027aea239f69b569105beabb2ea461b6582abbec0cc28e12c4d9c9108fad26c7" exitCode=0 Jul 15 21:30:53.265194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:53.264842 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerDied","Data":"027aea239f69b569105beabb2ea461b6582abbec0cc28e12c4d9c9108fad26c7"} Jul 15 21:30:53.275394 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:53.275373 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:54.057564 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:54.057528 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:54.057743 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:54.057614 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9l9v7" podUID="fe5819ec-966e-4773-b30b-4e311e9e8f94" Jul 15 21:30:54.057904 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:54.057878 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:54.058024 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:54.058007 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:54.058063 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:54.058012 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tg8tn" podUID="b1d9bff4-d3f4-47ea-9951-48073e0519a9" Jul 15 21:30:54.058125 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:54.058089 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7qtb7" podUID="16cfb664-2ac4-4852-8f0b-85b1422267e6" Jul 15 21:30:55.271342 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.271309 2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-178.ec2.internal" event="NodeReady" Jul 15 21:30:55.271720 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.271461 2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 21:30:55.319065 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.319036 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4rp7g"] Jul 15 21:30:55.332575 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.332550 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-slzq9"] Jul 15 21:30:55.332736 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.332662 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.335156 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.335115 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 21:30:55.335156 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.335142 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 21:30:55.335334 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.335263 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5mj78\"" Jul 15 21:30:55.335417 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.335401 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 21:30:55.346492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.346472 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rp7g"] Jul 15 21:30:55.346492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.346502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-slzq9"] Jul 15 21:30:55.346650 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.346607 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.349207 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.349187 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 21:30:55.349432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.349316 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 21:30:55.349432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.349322 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 21:30:55.349569 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.349475 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 21:30:55.349620 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.349570 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-w8f6x\"" Jul 15 21:30:55.417853 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.417825 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wv84r"] Jul 15 21:30:55.432953 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.432876 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wv84r"] Jul 15 21:30:55.433096 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.433006 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.435504 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.435484 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 21:30:55.435622 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.435493 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 21:30:55.435691 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.435641 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c96dg\"" Jul 15 21:30:55.447682 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/80c38c57-d37a-474d-911a-af4fb6182de6-data-volume\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.447798 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447705 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkt6p\" (UniqueName: \"kubernetes.io/projected/4fc95afb-8b40-4b9e-8fef-74cc549460ea-kube-api-access-zkt6p\") pod \"ingress-canary-4rp7g\" (UID: \"4fc95afb-8b40-4b9e-8fef-74cc549460ea\") " pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.447798 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447740 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5n8p\" (UniqueName: \"kubernetes.io/projected/80c38c57-d37a-474d-911a-af4fb6182de6-kube-api-access-b5n8p\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.447908 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc95afb-8b40-4b9e-8fef-74cc549460ea-cert\") pod \"ingress-canary-4rp7g\" (UID: \"4fc95afb-8b40-4b9e-8fef-74cc549460ea\") " pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.447908 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447872 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80c38c57-d37a-474d-911a-af4fb6182de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.448006 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447935 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/80c38c57-d37a-474d-911a-af4fb6182de6-crio-socket\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.448006 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.447976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80c38c57-d37a-474d-911a-af4fb6182de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.548734 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548661 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkt6p\" (UniqueName: \"kubernetes.io/projected/4fc95afb-8b40-4b9e-8fef-74cc549460ea-kube-api-access-zkt6p\") pod \"ingress-canary-4rp7g\" (UID: \"4fc95afb-8b40-4b9e-8fef-74cc549460ea\") " pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.548734 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5n8p\" (UniqueName: \"kubernetes.io/projected/80c38c57-d37a-474d-911a-af4fb6182de6-kube-api-access-b5n8p\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.548947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548752 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc95afb-8b40-4b9e-8fef-74cc549460ea-cert\") pod \"ingress-canary-4rp7g\" (UID: \"4fc95afb-8b40-4b9e-8fef-74cc549460ea\") " pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.548947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548780 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc69e67e-2596-4e81-b465-68ea1ef9fa78-config-volume\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.548947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548803 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc69e67e-2596-4e81-b465-68ea1ef9fa78-tmp-dir\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.548947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548835 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80c38c57-d37a-474d-911a-af4fb6182de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.548947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548879 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/80c38c57-d37a-474d-911a-af4fb6182de6-crio-socket\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.548947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548904 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mqpw\" (UniqueName: \"kubernetes.io/projected/bc69e67e-2596-4e81-b465-68ea1ef9fa78-kube-api-access-5mqpw\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.549238 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.548952 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80c38c57-d37a-474d-911a-af4fb6182de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.549238 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.549025 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/80c38c57-d37a-474d-911a-af4fb6182de6-data-volume\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.549238 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.549084 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc69e67e-2596-4e81-b465-68ea1ef9fa78-metrics-tls\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.549375 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.549309 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/80c38c57-d37a-474d-911a-af4fb6182de6-crio-socket\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.549412 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.549372 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/80c38c57-d37a-474d-911a-af4fb6182de6-data-volume\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.549522 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.549499 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/80c38c57-d37a-474d-911a-af4fb6182de6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.553322 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.553298 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4fc95afb-8b40-4b9e-8fef-74cc549460ea-cert\") pod \"ingress-canary-4rp7g\" (UID: \"4fc95afb-8b40-4b9e-8fef-74cc549460ea\") " pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.556701 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.556683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5n8p\" (UniqueName: \"kubernetes.io/projected/80c38c57-d37a-474d-911a-af4fb6182de6-kube-api-access-b5n8p\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.561584 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.561562 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/80c38c57-d37a-474d-911a-af4fb6182de6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-slzq9\" (UID: \"80c38c57-d37a-474d-911a-af4fb6182de6\") " pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.567546 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.567524 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkt6p\" (UniqueName: \"kubernetes.io/projected/4fc95afb-8b40-4b9e-8fef-74cc549460ea-kube-api-access-zkt6p\") pod \"ingress-canary-4rp7g\" (UID: \"4fc95afb-8b40-4b9e-8fef-74cc549460ea\") " pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.643079 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.643054 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4rp7g" Jul 15 21:30:55.650361 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.650337 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mqpw\" (UniqueName: \"kubernetes.io/projected/bc69e67e-2596-4e81-b465-68ea1ef9fa78-kube-api-access-5mqpw\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.650457 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.650384 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc69e67e-2596-4e81-b465-68ea1ef9fa78-metrics-tls\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.650521 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.650504 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc69e67e-2596-4e81-b465-68ea1ef9fa78-config-volume\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.650555 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.650538 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc69e67e-2596-4e81-b465-68ea1ef9fa78-tmp-dir\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.650826 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.650805 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc69e67e-2596-4e81-b465-68ea1ef9fa78-tmp-dir\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.650957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.650941 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc69e67e-2596-4e81-b465-68ea1ef9fa78-config-volume\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.652502 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.652482 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc69e67e-2596-4e81-b465-68ea1ef9fa78-metrics-tls\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.656452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.656433 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-slzq9" Jul 15 21:30:55.657584 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.657568 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mqpw\" (UniqueName: \"kubernetes.io/projected/bc69e67e-2596-4e81-b465-68ea1ef9fa78-kube-api-access-5mqpw\") pod \"dns-default-wv84r\" (UID: \"bc69e67e-2596-4e81-b465-68ea1ef9fa78\") " pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.750699 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.750319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wv84r" Jul 15 21:30:55.751879 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.751131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:55.751879 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.751417 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:55.751879 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.751480 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs podName:16cfb664-2ac4-4852-8f0b-85b1422267e6 nodeName:}" failed. No retries permitted until 2026-07-15 21:31:27.751459051 +0000 UTC m=+84.332056074 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs") pod "network-metrics-daemon-7qtb7" (UID: "16cfb664-2ac4-4852-8f0b-85b1422267e6") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 21:30:55.845841 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.845814 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-slzq9"] Jul 15 21:30:55.848344 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.848320 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4rp7g"] Jul 15 21:30:55.852564 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.852541 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:55.852738 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.852717 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 21:30:55.852807 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.852745 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 21:30:55.852807 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.852758 2572 projected.go:196] Error preparing data for projected volume kube-api-access-th4zc for pod openshift-network-diagnostics/network-check-target-9l9v7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:55.852900 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.852814 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc podName:fe5819ec-966e-4773-b30b-4e311e9e8f94 nodeName:}" failed. No retries permitted until 2026-07-15 21:31:27.852796683 +0000 UTC m=+84.433393708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-th4zc" (UniqueName: "kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc") pod "network-check-target-9l9v7" (UID: "fe5819ec-966e-4773-b30b-4e311e9e8f94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 21:30:55.953146 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:55.953113 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:55.953310 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.953279 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:55.953366 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:30:55.953358 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret podName:b1d9bff4-d3f4-47ea-9951-48073e0519a9 nodeName:}" failed. No retries permitted until 2026-07-15 21:31:27.953338339 +0000 UTC m=+84.533935365 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret") pod "global-pull-secret-syncer-tg8tn" (UID: "b1d9bff4-d3f4-47ea-9951-48073e0519a9") : object "kube-system"/"original-pull-secret" not registered Jul 15 21:30:56.057012 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.056923 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:30:56.057180 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.057039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:30:56.057343 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.057326 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:30:56.061452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.060440 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 21:30:56.061452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.060508 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 21:30:56.061452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.060904 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 21:30:56.061452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.061182 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 21:30:56.061452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.061239 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-75svw\"" Jul 15 21:30:56.061452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.061269 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8nwm9\"" Jul 15 21:30:56.595524 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.595495 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:56.648952 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:56.648914 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:30:57.185525 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:57.185496 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c38c57_d37a_474d_911a_af4fb6182de6.slice/crio-a9bb936cc36a21c9d2357a339fae25aaf4503ea4123143fcd19b495f11ea164c WatchSource:0}: Error finding container a9bb936cc36a21c9d2357a339fae25aaf4503ea4123143fcd19b495f11ea164c: Status 404 returned error can't find the container with id a9bb936cc36a21c9d2357a339fae25aaf4503ea4123143fcd19b495f11ea164c Jul 15 21:30:57.186790 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:57.186750 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc95afb_8b40_4b9e_8fef_74cc549460ea.slice/crio-12f8cec4c042fdce214b12ba9f744ef593738a0e5a7c459f7b02ecfe3c85ad03 WatchSource:0}: Error finding container 12f8cec4c042fdce214b12ba9f744ef593738a0e5a7c459f7b02ecfe3c85ad03: Status 404 returned error can't find the container with id 12f8cec4c042fdce214b12ba9f744ef593738a0e5a7c459f7b02ecfe3c85ad03 Jul 15 21:30:57.274843 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:57.274676 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzq9" event={"ID":"80c38c57-d37a-474d-911a-af4fb6182de6","Type":"ContainerStarted","Data":"a9bb936cc36a21c9d2357a339fae25aaf4503ea4123143fcd19b495f11ea164c"} Jul 15 21:30:57.275835 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:57.275805 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rp7g" event={"ID":"4fc95afb-8b40-4b9e-8fef-74cc549460ea","Type":"ContainerStarted","Data":"12f8cec4c042fdce214b12ba9f744ef593738a0e5a7c459f7b02ecfe3c85ad03"} Jul 15 21:30:57.308900 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:57.308878 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wv84r"] Jul 15 21:30:57.311767 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:30:57.311741 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc69e67e_2596_4e81_b465_68ea1ef9fa78.slice/crio-396bea27ca1ef91de53eda4d25949b43e39c88168e8c4a855a8152ab458daf31 WatchSource:0}: Error finding container 396bea27ca1ef91de53eda4d25949b43e39c88168e8c4a855a8152ab458daf31: Status 404 returned error can't find the container with id 396bea27ca1ef91de53eda4d25949b43e39c88168e8c4a855a8152ab458daf31 Jul 15 21:30:58.285511 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:58.285476 2572 generic.go:356] "Generic (PLEG): container finished" podID="bae5c93e-952f-4779-9030-8d96e59ced89" containerID="0d2915c82e566df682e2e957084f3d046637ffe294df947d2a8ee1af7d083659" exitCode=0 Jul 15 21:30:58.286264 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:58.285586 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerDied","Data":"0d2915c82e566df682e2e957084f3d046637ffe294df947d2a8ee1af7d083659"} Jul 15 21:30:58.287388 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:58.287360 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wv84r" event={"ID":"bc69e67e-2596-4e81-b465-68ea1ef9fa78","Type":"ContainerStarted","Data":"396bea27ca1ef91de53eda4d25949b43e39c88168e8c4a855a8152ab458daf31"} Jul 15 21:30:59.295195 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:59.294891 2572 generic.go:356] "Generic (PLEG): container finished" podID="bae5c93e-952f-4779-9030-8d96e59ced89" containerID="f93bde259973f2399255b268b422f9d9f3e1d51964f2dc3eaf1f034baeb69a56" exitCode=0 Jul 15 21:30:59.295195 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:30:59.294953 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerDied","Data":"f93bde259973f2399255b268b422f9d9f3e1d51964f2dc3eaf1f034baeb69a56"} Jul 15 21:31:00.216101 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:00.216065 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:00.270881 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:00.270843 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:00.397966 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:00.397900 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:03.451818 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:03.451779 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:03.523469 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:03.523433 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:04.310630 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.310528 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wv84r" event={"ID":"bc69e67e-2596-4e81-b465-68ea1ef9fa78","Type":"ContainerStarted","Data":"679cfd125af9aa2f6c21c3c59c198ca01b00eef73e9cd45be23253b7085fce38"} Jul 15 21:31:04.310630 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.310596 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wv84r" event={"ID":"bc69e67e-2596-4e81-b465-68ea1ef9fa78","Type":"ContainerStarted","Data":"f2d093f3a3832bf50331f20440722aa1afa5c13c937a7ad557cc7fd5607fa200"} Jul 15 21:31:04.310630 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.310625 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wv84r" Jul 15 21:31:04.312000 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.311971 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4rp7g" event={"ID":"4fc95afb-8b40-4b9e-8fef-74cc549460ea","Type":"ContainerStarted","Data":"93e661bfcfa515ee11b3cd159be216d2c84a79954afd07853c79b20fce834018"} Jul 15 21:31:04.315088 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.315059 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" event={"ID":"bae5c93e-952f-4779-9030-8d96e59ced89","Type":"ContainerStarted","Data":"0bd723ff970557c738066913c4bb61e43204c1fe4d7fbe6a0a0f7c5e2da354bb"} Jul 15 21:31:04.316423 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.316402 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzq9" event={"ID":"80c38c57-d37a-474d-911a-af4fb6182de6","Type":"ContainerStarted","Data":"cbd8dd98b908f0222a2b7d02aec06773dd1dc23b2180c60db60a1ddea3adbe4c"} Jul 15 21:31:04.326007 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.325969 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wv84r" podStartSLOduration=2.948283482 podStartE2EDuration="9.32595852s" totalImagesPullingTime="6.377675038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:04.325362477 +0000 UTC m=+60.905959519" watchObservedRunningTime="2026-07-15 21:31:04.32595852 +0000 UTC m=+60.906555546" Jul 15 21:31:04.339544 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.339509 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4rp7g" podStartSLOduration=2.836833156 podStartE2EDuration="9.339495185s" totalImagesPullingTime="6.502662029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:04.339307875 +0000 UTC m=+60.919904905" watchObservedRunningTime="2026-07-15 21:31:04.339495185 +0000 UTC m=+60.920092211" Jul 15 21:31:04.361311 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.361272 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tfzw6" podStartSLOduration=11.639795977 podStartE2EDuration="41.36126028s" totalImagesPullingTime="29.721464303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:04.360030974 +0000 UTC m=+60.940628024" watchObservedRunningTime="2026-07-15 21:31:04.36126028 +0000 UTC m=+60.941857307" Jul 15 21:31:04.479700 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.479670 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:04.531928 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:04.531907 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:06.904771 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:06.904734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:07.006070 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:07.006033 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:08.330703 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:08.330669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzq9" event={"ID":"80c38c57-d37a-474d-911a-af4fb6182de6","Type":"ContainerStarted","Data":"da033698c9c90b512a78228acf5c74e581efcdc2e03ed2f66ce0e23d71641c59"} Jul 15 21:31:14.321552 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:14.321520 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wv84r" Jul 15 21:31:14.351739 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:14.351694 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-slzq9" event={"ID":"80c38c57-d37a-474d-911a-af4fb6182de6","Type":"ContainerStarted","Data":"c5e8e866e202494ea918d56dd7070d7715ada63af6dbedd862d975c7271d341d"} Jul 15 21:31:14.369083 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:14.369032 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-slzq9" podStartSLOduration=2.9910529009999998 podStartE2EDuration="19.36901597s" totalImagesPullingTime="16.377963069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:14.368102308 +0000 UTC m=+70.948699356" watchObservedRunningTime="2026-07-15 21:31:14.36901597 +0000 UTC m=+70.949613001" Jul 15 21:31:18.448441 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:18.448397 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:18.499742 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:18.499714 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:22.276589 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:22.276559 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lnzxm" Jul 15 21:31:22.380451 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:22.380423 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:22.483232 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:22.483203 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:22.750050 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:22.750016 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:27.805045 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.804931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:31:27.807818 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.807799 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 21:31:27.818872 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.818845 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16cfb664-2ac4-4852-8f0b-85b1422267e6-metrics-certs\") pod \"network-metrics-daemon-7qtb7\" (UID: \"16cfb664-2ac4-4852-8f0b-85b1422267e6\") " pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:31:27.881066 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.881033 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8nwm9\"" Jul 15 21:31:27.889066 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.889031 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7qtb7" Jul 15 21:31:27.906350 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.906310 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:31:27.908794 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.908776 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 21:31:27.919426 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.919393 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 21:31:27.929633 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:27.929610 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th4zc\" (UniqueName: \"kubernetes.io/projected/fe5819ec-966e-4773-b30b-4e311e9e8f94-kube-api-access-th4zc\") pod \"network-check-target-9l9v7\" (UID: \"fe5819ec-966e-4773-b30b-4e311e9e8f94\") " pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:31:28.007563 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.007523 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:31:28.010877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.010847 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7qtb7"] Jul 15 21:31:28.011002 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.010972 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 21:31:28.013754 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:31:28.013733 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16cfb664_2ac4_4852_8f0b_85b1422267e6.slice/crio-375682472c1017bec5cc69772e6efe8f348579493a6ee8baf8127773e84e75e0 WatchSource:0}: Error finding container 375682472c1017bec5cc69772e6efe8f348579493a6ee8baf8127773e84e75e0: Status 404 returned error can't find the container with id 375682472c1017bec5cc69772e6efe8f348579493a6ee8baf8127773e84e75e0 Jul 15 21:31:28.020049 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.020027 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b1d9bff4-d3f4-47ea-9951-48073e0519a9-original-pull-secret\") pod \"global-pull-secret-syncer-tg8tn\" (UID: \"b1d9bff4-d3f4-47ea-9951-48073e0519a9\") " pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:31:28.173571 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.173490 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-75svw\"" Jul 15 21:31:28.181383 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.181360 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:31:28.184625 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.184608 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tg8tn" Jul 15 21:31:28.310343 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.310313 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tg8tn"] Jul 15 21:31:28.313406 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:31:28.313383 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d9bff4_d3f4_47ea_9951_48073e0519a9.slice/crio-0c8cb823b060fcd5cc7f61b9e21f869bbee45467aabf77fb703ffd1d9b84c55a WatchSource:0}: Error finding container 0c8cb823b060fcd5cc7f61b9e21f869bbee45467aabf77fb703ffd1d9b84c55a: Status 404 returned error can't find the container with id 0c8cb823b060fcd5cc7f61b9e21f869bbee45467aabf77fb703ffd1d9b84c55a Jul 15 21:31:28.324940 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.324915 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9l9v7"] Jul 15 21:31:28.327616 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:31:28.327594 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5819ec_966e_4773_b30b_4e311e9e8f94.slice/crio-153be65d8c6f0bf0fdd9cc39f534df0bc12f699ac92ba3a0ac7b833619502796 WatchSource:0}: Error finding container 153be65d8c6f0bf0fdd9cc39f534df0bc12f699ac92ba3a0ac7b833619502796: Status 404 returned error can't find the container with id 153be65d8c6f0bf0fdd9cc39f534df0bc12f699ac92ba3a0ac7b833619502796 Jul 15 21:31:28.395871 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.395833 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qtb7" event={"ID":"16cfb664-2ac4-4852-8f0b-85b1422267e6","Type":"ContainerStarted","Data":"375682472c1017bec5cc69772e6efe8f348579493a6ee8baf8127773e84e75e0"} Jul 15 21:31:28.397323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.397293 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9l9v7" event={"ID":"fe5819ec-966e-4773-b30b-4e311e9e8f94","Type":"ContainerStarted","Data":"153be65d8c6f0bf0fdd9cc39f534df0bc12f699ac92ba3a0ac7b833619502796"} Jul 15 21:31:28.398465 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:28.398435 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tg8tn" event={"ID":"b1d9bff4-d3f4-47ea-9951-48073e0519a9","Type":"ContainerStarted","Data":"0c8cb823b060fcd5cc7f61b9e21f869bbee45467aabf77fb703ffd1d9b84c55a"} Jul 15 21:31:29.403998 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:29.403951 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qtb7" event={"ID":"16cfb664-2ac4-4852-8f0b-85b1422267e6","Type":"ContainerStarted","Data":"0add3eb960fea01c30acea7fd3c78fa2d6c426e6e3c5d6b6ea61d0e0f88f5c00"} Jul 15 21:31:29.404493 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:29.404021 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7qtb7" event={"ID":"16cfb664-2ac4-4852-8f0b-85b1422267e6","Type":"ContainerStarted","Data":"18d11b5005c76895a4a6ff968d67a7327d99539ddea6ef6ffa421d9ca36e6f45"} Jul 15 21:31:29.422581 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:29.422223 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7qtb7" podStartSLOduration=65.653936313 podStartE2EDuration="1m6.422204182s" totalImagesPullingTime="768.267869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:29.421190699 +0000 UTC m=+86.001787750" watchObservedRunningTime="2026-07-15 21:31:29.422204182 +0000 UTC m=+86.002801212" Jul 15 21:31:29.503664 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:29.503611 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:29.561482 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:29.561448 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:32.416249 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:32.416203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9l9v7" event={"ID":"fe5819ec-966e-4773-b30b-4e311e9e8f94","Type":"ContainerStarted","Data":"166770df5d270a26859d88e8373785b707fd2bffec7669671ae290ab42779998"} Jul 15 21:31:32.416697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:32.416406 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:31:32.432392 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:32.432335 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9l9v7" podStartSLOduration=66.209411567 podStartE2EDuration="1m9.432318754s" totalImagesPullingTime="3.222907187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:32.430846369 +0000 UTC m=+89.011443398" watchObservedRunningTime="2026-07-15 21:31:32.432318754 +0000 UTC m=+89.012915782" Jul 15 21:31:33.421313 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:33.421218 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tg8tn" event={"ID":"b1d9bff4-d3f4-47ea-9951-48073e0519a9","Type":"ContainerStarted","Data":"5d24729f27842d3e752e41bc4f8a68f6572e75cf14f9630605d14df6a2985396"} Jul 15 21:31:33.436226 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:33.436182 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tg8tn" podStartSLOduration=64.657704 podStartE2EDuration="1m9.436145759s" totalImagesPullingTime="4.778441759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 21:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:31:33.435037891 +0000 UTC m=+90.015634920" watchObservedRunningTime="2026-07-15 21:31:33.436145759 +0000 UTC m=+90.016742788" Jul 15 21:31:51.972443 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:51.972405 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:52.022962 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:52.022934 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:31:57.946874 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.946841 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:31:57.949049 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.949033 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:57.951645 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.951620 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 21:31:57.951764 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.951623 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 21:31:57.951897 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.951879 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-47mtv\"" Jul 15 21:31:57.952067 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.952046 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 21:31:57.952202 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.952137 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 21:31:57.952202 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.952046 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 21:31:57.952202 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.952104 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 21:31:57.952371 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.952141 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 21:31:57.952371 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.952112 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 21:31:57.958402 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.958385 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 21:31:57.963575 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:57.963554 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:31:58.006969 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.006932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.006969 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.006969 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02bfcb67-f844-449d-8682-9e2152580cbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.006995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007045 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02bfcb67-f844-449d-8682-9e2152580cbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007066 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007086 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-web-config\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007121 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02bfcb67-f844-449d-8682-9e2152580cbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007239 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02bfcb67-f844-449d-8682-9e2152580cbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007262 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w9x\" (UniqueName: \"kubernetes.io/projected/02bfcb67-f844-449d-8682-9e2152580cbf-kube-api-access-c7w9x\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007276 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02bfcb67-f844-449d-8682-9e2152580cbf-config-out\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.007415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.007305 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108624 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108581 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108624 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108622 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02bfcb67-f844-449d-8682-9e2152580cbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108643 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108688 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02bfcb67-f844-449d-8682-9e2152580cbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108712 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108737 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-web-config\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108768 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02bfcb67-f844-449d-8682-9e2152580cbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108828 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02bfcb67-f844-449d-8682-9e2152580cbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.108858 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108853 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w9x\" (UniqueName: \"kubernetes.io/projected/02bfcb67-f844-449d-8682-9e2152580cbf-kube-api-access-c7w9x\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.109313 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108868 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02bfcb67-f844-449d-8682-9e2152580cbf-config-out\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.109313 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.108905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.109718 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.109688 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02bfcb67-f844-449d-8682-9e2152580cbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.110226 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.110143 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/02bfcb67-f844-449d-8682-9e2152580cbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.110625 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.110596 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/02bfcb67-f844-449d-8682-9e2152580cbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.111861 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.111820 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.112408 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.112383 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/02bfcb67-f844-449d-8682-9e2152580cbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.112513 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.112393 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.112834 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.112807 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-web-config\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.112914 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.112831 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.112973 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.112939 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.113068 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.113047 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/02bfcb67-f844-449d-8682-9e2152580cbf-config-out\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.113146 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.113129 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.113794 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.113774 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/02bfcb67-f844-449d-8682-9e2152580cbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.117023 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.117003 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w9x\" (UniqueName: \"kubernetes.io/projected/02bfcb67-f844-449d-8682-9e2152580cbf-kube-api-access-c7w9x\") pod \"alertmanager-main-0\" (UID: \"02bfcb67-f844-449d-8682-9e2152580cbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.258463 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.258363 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 21:31:58.389885 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.389861 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 21:31:58.392314 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:31:58.392288 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bfcb67_f844_449d_8682_9e2152580cbf.slice/crio-091a45f298f01325e4a395ee8775a1d28518c9c519a22363d46eb245c9f6c415 WatchSource:0}: Error finding container 091a45f298f01325e4a395ee8775a1d28518c9c519a22363d46eb245c9f6c415: Status 404 returned error can't find the container with id 091a45f298f01325e4a395ee8775a1d28518c9c519a22363d46eb245c9f6c415 Jul 15 21:31:58.498050 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:31:58.498016 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"091a45f298f01325e4a395ee8775a1d28518c9c519a22363d46eb245c9f6c415"} Jul 15 21:32:00.505233 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:00.505205 2572 generic.go:356] "Generic (PLEG): container finished" podID="02bfcb67-f844-449d-8682-9e2152580cbf" containerID="40dbccd68e6a61e7c4c8b84eecb5aec0524f8698e45252290b7ae0237b0be90a" exitCode=0 Jul 15 21:32:00.505703 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:00.505288 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerDied","Data":"40dbccd68e6a61e7c4c8b84eecb5aec0524f8698e45252290b7ae0237b0be90a"} Jul 15 21:32:02.513888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:02.513833 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"1a17ac44bc53b5808222fb9159d50f37eeaedd65a3492fd7a32a66a89462a907"} Jul 15 21:32:02.514156 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:02.513928 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"15358ecc6cb58cdcfbd281f4b726a4cfbf9be830e1246c1e6e572bad436e73b2"} Jul 15 21:32:03.423950 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:03.423917 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9l9v7" Jul 15 21:32:03.520236 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:03.520203 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"943e6b46ad76a002c60feb22ef51267d353e6af853b8d766827d165d6e0e7dff"} Jul 15 21:32:03.520752 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:03.520273 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"ef817bf53bb5b61e704c88b2df5fe4000a1e273764e18b50e362ef4dff6b0bf5"} Jul 15 21:32:03.520752 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:03.520299 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"e2760def82d44f7c9fdee6977735a3c4ed64ecdc8f8573950c9779c8e49a1906"} Jul 15 21:32:03.879376 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:03.879345 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:03.934975 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:03.934951 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:05.530194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:05.530142 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"02bfcb67-f844-449d-8682-9e2152580cbf","Type":"ContainerStarted","Data":"94500e1571449f0a24df15b2b7d96b8c165914e4c47eccbac59fa4094223fcd5"} Jul 15 21:32:05.558089 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:05.558042 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.880402294 podStartE2EDuration="8.558025881s" totalImagesPullingTime="5.660462912s" totalInitContainerRuntime="17.160675ms" isStatefulPod=false podCreationTimestamp="2026-07-15 21:31:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:32:05.556115228 +0000 UTC m=+122.136712272" watchObservedRunningTime="2026-07-15 21:32:05.558025881 +0000 UTC m=+122.138622912" Jul 15 21:32:15.398599 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:15.398561 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:15.446677 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:15.446650 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:15.541414 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:15.541387 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:15.642497 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:15.642469 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:16.136254 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:16.136226 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:16.185719 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:16.185695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:18.453500 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:18.453467 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:18.555721 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:18.555695 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:18.604786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:18.604757 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:20.034859 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:20.034821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:26.123814 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:26.123779 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:31.621768 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:31.621734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:32:37.802818 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:32:37.802754 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:33:38.436566 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:38.436529 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:33:38.535337 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:38.535308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:33:39.001913 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.001881 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p"] Jul 15 21:33:39.004055 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.004039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.006561 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.006532 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 21:33:39.007481 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.007465 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-psgb9\"" Jul 15 21:33:39.007594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.007471 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 21:33:39.016543 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.016522 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p"] Jul 15 21:33:39.164121 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.164093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tk97\" (UniqueName: \"kubernetes.io/projected/497e6da8-3595-444a-991a-0f9b67d0f666-kube-api-access-9tk97\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.164272 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.164127 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.164272 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.164150 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.265268 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.265197 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.265393 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.265266 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tk97\" (UniqueName: \"kubernetes.io/projected/497e6da8-3595-444a-991a-0f9b67d0f666-kube-api-access-9tk97\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.265393 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.265301 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.265534 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.265514 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.265620 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.265604 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.273348 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.273329 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tk97\" (UniqueName: \"kubernetes.io/projected/497e6da8-3595-444a-991a-0f9b67d0f666-kube-api-access-9tk97\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.312334 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.312310 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:33:39.429309 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.429272 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p"] Jul 15 21:33:39.432886 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:33:39.432854 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497e6da8_3595_444a_991a_0f9b67d0f666.slice/crio-322f3d64e5e6c71ff5b2e36670d347884152c2d427015dc3506af51ce26a9e3a WatchSource:0}: Error finding container 322f3d64e5e6c71ff5b2e36670d347884152c2d427015dc3506af51ce26a9e3a: Status 404 returned error can't find the container with id 322f3d64e5e6c71ff5b2e36670d347884152c2d427015dc3506af51ce26a9e3a Jul 15 21:33:39.817113 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:39.817082 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" event={"ID":"497e6da8-3595-444a-991a-0f9b67d0f666","Type":"ContainerStarted","Data":"322f3d64e5e6c71ff5b2e36670d347884152c2d427015dc3506af51ce26a9e3a"} Jul 15 21:33:47.842896 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:47.842865 2572 generic.go:356] "Generic (PLEG): container finished" podID="497e6da8-3595-444a-991a-0f9b67d0f666" containerID="5d19003cc45a0f067a40b3230cd415410e0a242d76fac259ad6281b91afd599a" exitCode=0 Jul 15 21:33:47.843434 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:47.842923 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" event={"ID":"497e6da8-3595-444a-991a-0f9b67d0f666","Type":"ContainerDied","Data":"5d19003cc45a0f067a40b3230cd415410e0a242d76fac259ad6281b91afd599a"} Jul 15 21:33:50.854038 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:50.854001 2572 generic.go:356] "Generic (PLEG): container finished" podID="497e6da8-3595-444a-991a-0f9b67d0f666" containerID="99e7da926b2118652f29d89a0fb44b424b9e5f79f79f3d16cb604e0824bb4b9f" exitCode=0 Jul 15 21:33:50.854429 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:50.854074 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" event={"ID":"497e6da8-3595-444a-991a-0f9b67d0f666","Type":"ContainerDied","Data":"99e7da926b2118652f29d89a0fb44b424b9e5f79f79f3d16cb604e0824bb4b9f"} Jul 15 21:33:58.886127 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:58.886088 2572 generic.go:356] "Generic (PLEG): container finished" podID="497e6da8-3595-444a-991a-0f9b67d0f666" containerID="2511ed14f228171ecd4755b118258b20870c5dd7f0e2ba2158e2d8dec268b450" exitCode=0 Jul 15 21:33:58.886594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:33:58.886150 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" event={"ID":"497e6da8-3595-444a-991a-0f9b67d0f666","Type":"ContainerDied","Data":"2511ed14f228171ecd4755b118258b20870c5dd7f0e2ba2158e2d8dec268b450"} Jul 15 21:34:00.008130 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.008106 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:34:00.108818 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.108795 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tk97\" (UniqueName: \"kubernetes.io/projected/497e6da8-3595-444a-991a-0f9b67d0f666-kube-api-access-9tk97\") pod \"497e6da8-3595-444a-991a-0f9b67d0f666\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " Jul 15 21:34:00.108915 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.108839 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-util\") pod \"497e6da8-3595-444a-991a-0f9b67d0f666\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " Jul 15 21:34:00.108915 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.108882 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-bundle\") pod \"497e6da8-3595-444a-991a-0f9b67d0f666\" (UID: \"497e6da8-3595-444a-991a-0f9b67d0f666\") " Jul 15 21:34:00.109536 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.109512 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-bundle" (OuterVolumeSpecName: "bundle") pod "497e6da8-3595-444a-991a-0f9b67d0f666" (UID: "497e6da8-3595-444a-991a-0f9b67d0f666"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:34:00.111036 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.111013 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497e6da8-3595-444a-991a-0f9b67d0f666-kube-api-access-9tk97" (OuterVolumeSpecName: "kube-api-access-9tk97") pod "497e6da8-3595-444a-991a-0f9b67d0f666" (UID: "497e6da8-3595-444a-991a-0f9b67d0f666"). InnerVolumeSpecName "kube-api-access-9tk97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:34:00.114272 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.114245 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-util" (OuterVolumeSpecName: "util") pod "497e6da8-3595-444a-991a-0f9b67d0f666" (UID: "497e6da8-3595-444a-991a-0f9b67d0f666"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:34:00.209526 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.209491 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:00.209526 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.209524 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tk97\" (UniqueName: \"kubernetes.io/projected/497e6da8-3595-444a-991a-0f9b67d0f666-kube-api-access-9tk97\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:00.209699 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.209536 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497e6da8-3595-444a-991a-0f9b67d0f666-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:00.894064 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.894030 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" event={"ID":"497e6da8-3595-444a-991a-0f9b67d0f666","Type":"ContainerDied","Data":"322f3d64e5e6c71ff5b2e36670d347884152c2d427015dc3506af51ce26a9e3a"} Jul 15 21:34:00.894253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.894077 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540p2b2p" Jul 15 21:34:00.894253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:00.894078 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="322f3d64e5e6c71ff5b2e36670d347884152c2d427015dc3506af51ce26a9e3a" Jul 15 21:34:05.034036 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:05.034006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:05.837520 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:05.837486 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:06.992973 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.992934 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh"] Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993299 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="pull" Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993317 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="pull" Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993369 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="util" Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993379 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="util" Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993388 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="extract" Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993395 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="extract" Jul 15 21:34:06.993472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.993462 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="497e6da8-3595-444a-991a-0f9b67d0f666" containerName="extract" Jul 15 21:34:06.996715 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.996693 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:06.999344 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.999325 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 15 21:34:06.999344 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.999332 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7pvch\"" Jul 15 21:34:06.999562 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:06.999550 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:34:07.009973 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.009951 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh"] Jul 15 21:34:07.056272 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.056242 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7690f779-3255-4980-904d-cf4ea6758729-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-q6gwh\" (UID: \"7690f779-3255-4980-904d-cf4ea6758729\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.056429 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.056304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx4lq\" (UniqueName: \"kubernetes.io/projected/7690f779-3255-4980-904d-cf4ea6758729-kube-api-access-vx4lq\") pod \"cert-manager-operator-controller-manager-55c444bf86-q6gwh\" (UID: \"7690f779-3255-4980-904d-cf4ea6758729\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.156926 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.156899 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx4lq\" (UniqueName: \"kubernetes.io/projected/7690f779-3255-4980-904d-cf4ea6758729-kube-api-access-vx4lq\") pod \"cert-manager-operator-controller-manager-55c444bf86-q6gwh\" (UID: \"7690f779-3255-4980-904d-cf4ea6758729\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.157035 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.156934 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7690f779-3255-4980-904d-cf4ea6758729-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-q6gwh\" (UID: \"7690f779-3255-4980-904d-cf4ea6758729\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.157285 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.157268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7690f779-3255-4980-904d-cf4ea6758729-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-q6gwh\" (UID: \"7690f779-3255-4980-904d-cf4ea6758729\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.165084 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.165058 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx4lq\" (UniqueName: \"kubernetes.io/projected/7690f779-3255-4980-904d-cf4ea6758729-kube-api-access-vx4lq\") pod \"cert-manager-operator-controller-manager-55c444bf86-q6gwh\" (UID: \"7690f779-3255-4980-904d-cf4ea6758729\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.305401 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.305322 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" Jul 15 21:34:07.428431 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.428257 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh"] Jul 15 21:34:07.431151 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:07.431121 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7690f779_3255_4980_904d_cf4ea6758729.slice/crio-57ef9ea827cff0c7e613c3886ab7b5ec5b059f0499b220ffa5c596a41462cab3 WatchSource:0}: Error finding container 57ef9ea827cff0c7e613c3886ab7b5ec5b059f0499b220ffa5c596a41462cab3: Status 404 returned error can't find the container with id 57ef9ea827cff0c7e613c3886ab7b5ec5b059f0499b220ffa5c596a41462cab3 Jul 15 21:34:07.917426 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:07.917393 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" event={"ID":"7690f779-3255-4980-904d-cf4ea6758729","Type":"ContainerStarted","Data":"57ef9ea827cff0c7e613c3886ab7b5ec5b059f0499b220ffa5c596a41462cab3"} Jul 15 21:34:08.633704 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:08.633654 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:09.926367 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:09.926333 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" event={"ID":"7690f779-3255-4980-904d-cf4ea6758729","Type":"ContainerStarted","Data":"0a12022e3a5211b84ab48d41577a8a1da27dc16cab0be6713f7e1dd7a2b8575d"} Jul 15 21:34:09.949317 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:09.949261 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-q6gwh" podStartSLOduration=2.484857275 podStartE2EDuration="3.94924703s" totalImagesPullingTime="1.464389755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:34:09.94712372 +0000 UTC m=+246.527720761" watchObservedRunningTime="2026-07-15 21:34:09.94924703 +0000 UTC m=+246.529844059" Jul 15 21:34:10.332725 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.332697 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:10.436789 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.436758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:10.669035 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.668962 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:10.767547 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.767516 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:10.861469 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.861440 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7"] Jul 15 21:34:10.863370 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.863355 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:10.865994 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.865977 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 21:34:10.866088 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.866075 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 21:34:10.866986 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.866971 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-psgb9\"" Jul 15 21:34:10.873992 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.873971 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7"] Jul 15 21:34:10.986888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.986863 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:10.987278 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.986931 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:10.987278 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:10.986954 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv56n\" (UniqueName: \"kubernetes.io/projected/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-kube-api-access-zv56n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.087771 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.087744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.087960 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.087797 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv56n\" (UniqueName: \"kubernetes.io/projected/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-kube-api-access-zv56n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.087960 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.087832 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.088273 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.088252 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.088323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.088281 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.095714 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.095696 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv56n\" (UniqueName: \"kubernetes.io/projected/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-kube-api-access-zv56n\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.172692 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.172652 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:11.294192 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.294142 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7"] Jul 15 21:34:11.297271 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:11.297249 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fea1bc8_65e7_497e_8f5b_e65d57eaf531.slice/crio-ff86c4cbdb51196577b414387e86cd0eabc24c4c86e7269a1e04eeed32ffbcf7 WatchSource:0}: Error finding container ff86c4cbdb51196577b414387e86cd0eabc24c4c86e7269a1e04eeed32ffbcf7: Status 404 returned error can't find the container with id ff86c4cbdb51196577b414387e86cd0eabc24c4c86e7269a1e04eeed32ffbcf7 Jul 15 21:34:11.935061 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.935027 2572 generic.go:356] "Generic (PLEG): container finished" podID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerID="4726dbb524c4e17ec470ef2bf35bd509c4f1a9c6d48bdb0cebee280a189f7323" exitCode=0 Jul 15 21:34:11.935283 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.935084 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" event={"ID":"1fea1bc8-65e7-497e-8f5b-e65d57eaf531","Type":"ContainerDied","Data":"4726dbb524c4e17ec470ef2bf35bd509c4f1a9c6d48bdb0cebee280a189f7323"} Jul 15 21:34:11.935283 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:11.935120 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" event={"ID":"1fea1bc8-65e7-497e-8f5b-e65d57eaf531","Type":"ContainerStarted","Data":"ff86c4cbdb51196577b414387e86cd0eabc24c4c86e7269a1e04eeed32ffbcf7"} Jul 15 21:34:12.847225 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:12.847197 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:13.256498 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.256463 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:13.565897 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.565823 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6"] Jul 15 21:34:13.573620 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.573597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.576620 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.576598 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 21:34:13.577721 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.577665 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 21:34:13.577721 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.577678 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xpd7n\"" Jul 15 21:34:13.579549 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.579521 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6"] Jul 15 21:34:13.608761 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.608734 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf6ff1e-2e6b-43da-9496-db856c1997d4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ddwz6\" (UID: \"3bf6ff1e-2e6b-43da-9496-db856c1997d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.608875 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.608787 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g8r8\" (UniqueName: \"kubernetes.io/projected/3bf6ff1e-2e6b-43da-9496-db856c1997d4-kube-api-access-8g8r8\") pod \"cert-manager-cainjector-559cdd8c87-ddwz6\" (UID: \"3bf6ff1e-2e6b-43da-9496-db856c1997d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.710241 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.710208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf6ff1e-2e6b-43da-9496-db856c1997d4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ddwz6\" (UID: \"3bf6ff1e-2e6b-43da-9496-db856c1997d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.710413 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.710261 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g8r8\" (UniqueName: \"kubernetes.io/projected/3bf6ff1e-2e6b-43da-9496-db856c1997d4-kube-api-access-8g8r8\") pod \"cert-manager-cainjector-559cdd8c87-ddwz6\" (UID: \"3bf6ff1e-2e6b-43da-9496-db856c1997d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.719655 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.719629 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3bf6ff1e-2e6b-43da-9496-db856c1997d4-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-ddwz6\" (UID: \"3bf6ff1e-2e6b-43da-9496-db856c1997d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.719777 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.719723 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g8r8\" (UniqueName: \"kubernetes.io/projected/3bf6ff1e-2e6b-43da-9496-db856c1997d4-kube-api-access-8g8r8\") pod \"cert-manager-cainjector-559cdd8c87-ddwz6\" (UID: \"3bf6ff1e-2e6b-43da-9496-db856c1997d4\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:13.892431 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:13.892329 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" Jul 15 21:34:14.098236 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:14.098212 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6"] Jul 15 21:34:14.100455 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:14.100432 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf6ff1e_2e6b_43da_9496_db856c1997d4.slice/crio-8fd3f84669134cc9bba7f2bf081ae0e720c35003c896954eab74240f104dc1df WatchSource:0}: Error finding container 8fd3f84669134cc9bba7f2bf081ae0e720c35003c896954eab74240f104dc1df: Status 404 returned error can't find the container with id 8fd3f84669134cc9bba7f2bf081ae0e720c35003c896954eab74240f104dc1df Jul 15 21:34:14.951319 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:14.951284 2572 generic.go:356] "Generic (PLEG): container finished" podID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerID="127bea651ff3b224a39087e16a35c0e43d5657fbdbd207edd1dc7cb75e537afa" exitCode=0 Jul 15 21:34:14.951721 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:14.951351 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" event={"ID":"1fea1bc8-65e7-497e-8f5b-e65d57eaf531","Type":"ContainerDied","Data":"127bea651ff3b224a39087e16a35c0e43d5657fbdbd207edd1dc7cb75e537afa"} Jul 15 21:34:14.952548 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:14.952525 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" event={"ID":"3bf6ff1e-2e6b-43da-9496-db856c1997d4","Type":"ContainerStarted","Data":"8fd3f84669134cc9bba7f2bf081ae0e720c35003c896954eab74240f104dc1df"} Jul 15 21:34:15.432891 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:15.432808 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:15.959178 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:15.959124 2572 generic.go:356] "Generic (PLEG): container finished" podID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerID="eb261012ff424b3d671aaf842d98daaf35d3bcb193939fc6596a47bfc34e62d7" exitCode=0 Jul 15 21:34:15.959594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:15.959208 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" event={"ID":"1fea1bc8-65e7-497e-8f5b-e65d57eaf531","Type":"ContainerDied","Data":"eb261012ff424b3d671aaf842d98daaf35d3bcb193939fc6596a47bfc34e62d7"} Jul 15 21:34:16.198404 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:16.198368 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:16.424884 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:16.424808 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:17.291178 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.291143 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:17.343532 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.343513 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zv56n\" (UniqueName: \"kubernetes.io/projected/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-kube-api-access-zv56n\") pod \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " Jul 15 21:34:17.343632 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.343544 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-bundle\") pod \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " Jul 15 21:34:17.343632 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.343572 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-util\") pod \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\" (UID: \"1fea1bc8-65e7-497e-8f5b-e65d57eaf531\") " Jul 15 21:34:17.343906 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.343882 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-bundle" (OuterVolumeSpecName: "bundle") pod "1fea1bc8-65e7-497e-8f5b-e65d57eaf531" (UID: "1fea1bc8-65e7-497e-8f5b-e65d57eaf531"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:34:17.345626 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.345600 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-kube-api-access-zv56n" (OuterVolumeSpecName: "kube-api-access-zv56n") pod "1fea1bc8-65e7-497e-8f5b-e65d57eaf531" (UID: "1fea1bc8-65e7-497e-8f5b-e65d57eaf531"). InnerVolumeSpecName "kube-api-access-zv56n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:34:17.348270 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.348238 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-util" (OuterVolumeSpecName: "util") pod "1fea1bc8-65e7-497e-8f5b-e65d57eaf531" (UID: "1fea1bc8-65e7-497e-8f5b-e65d57eaf531"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:34:17.444947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.444911 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv56n\" (UniqueName: \"kubernetes.io/projected/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-kube-api-access-zv56n\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:17.444947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.444941 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:17.444947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.444951 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fea1bc8-65e7-497e-8f5b-e65d57eaf531-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:17.759595 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759558 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lz2mv"] Jul 15 21:34:17.759818 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759807 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="util" Jul 15 21:34:17.759867 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759819 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="util" Jul 15 21:34:17.759867 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759827 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="pull" Jul 15 21:34:17.759867 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759832 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="pull" Jul 15 21:34:17.759867 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759846 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="extract" Jul 15 21:34:17.759867 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759863 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="extract" Jul 15 21:34:17.760016 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.759907 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fea1bc8-65e7-497e-8f5b-e65d57eaf531" containerName="extract" Jul 15 21:34:17.762699 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.762684 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.765217 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.765186 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-f9ww8\"" Jul 15 21:34:17.770513 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.770495 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lz2mv"] Jul 15 21:34:17.847972 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.847937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g6hk\" (UniqueName: \"kubernetes.io/projected/df419a7c-0a7c-47aa-a3f8-de27031cd584-kube-api-access-5g6hk\") pod \"cert-manager-webhook-6486ffd997-lz2mv\" (UID: \"df419a7c-0a7c-47aa-a3f8-de27031cd584\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.848132 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.847986 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df419a7c-0a7c-47aa-a3f8-de27031cd584-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lz2mv\" (UID: \"df419a7c-0a7c-47aa-a3f8-de27031cd584\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.949086 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.949049 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g6hk\" (UniqueName: \"kubernetes.io/projected/df419a7c-0a7c-47aa-a3f8-de27031cd584-kube-api-access-5g6hk\") pod \"cert-manager-webhook-6486ffd997-lz2mv\" (UID: \"df419a7c-0a7c-47aa-a3f8-de27031cd584\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.949285 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.949107 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df419a7c-0a7c-47aa-a3f8-de27031cd584-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lz2mv\" (UID: \"df419a7c-0a7c-47aa-a3f8-de27031cd584\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.956837 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.956801 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df419a7c-0a7c-47aa-a3f8-de27031cd584-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lz2mv\" (UID: \"df419a7c-0a7c-47aa-a3f8-de27031cd584\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.956957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.956890 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g6hk\" (UniqueName: \"kubernetes.io/projected/df419a7c-0a7c-47aa-a3f8-de27031cd584-kube-api-access-5g6hk\") pod \"cert-manager-webhook-6486ffd997-lz2mv\" (UID: \"df419a7c-0a7c-47aa-a3f8-de27031cd584\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:17.968771 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.968742 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" event={"ID":"1fea1bc8-65e7-497e-8f5b-e65d57eaf531","Type":"ContainerDied","Data":"ff86c4cbdb51196577b414387e86cd0eabc24c4c86e7269a1e04eeed32ffbcf7"} Jul 15 21:34:17.968902 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.968800 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff86c4cbdb51196577b414387e86cd0eabc24c4c86e7269a1e04eeed32ffbcf7" Jul 15 21:34:17.968902 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.968749 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flgjk7" Jul 15 21:34:17.970220 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.970191 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" event={"ID":"3bf6ff1e-2e6b-43da-9496-db856c1997d4","Type":"ContainerStarted","Data":"f0af41f141321f2a96557666949f8d5f0662a6b762389fbf8570429cc9d99ab1"} Jul 15 21:34:17.984737 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:17.984695 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-ddwz6" podStartSLOduration=1.747253916 podStartE2EDuration="4.984680962s" totalImagesPullingTime="3.237427046s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:34:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:34:17.983471971 +0000 UTC m=+254.564069002" watchObservedRunningTime="2026-07-15 21:34:17.984680962 +0000 UTC m=+254.565277990" Jul 15 21:34:18.073917 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.073845 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:18.192267 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.192240 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lz2mv"] Jul 15 21:34:18.195492 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:18.195466 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf419a7c_0a7c_47aa_a3f8_de27031cd584.slice/crio-a37d80d28951aa9f7681d6624062a5df157d5e166f219739f46cd05199e183c8 WatchSource:0}: Error finding container a37d80d28951aa9f7681d6624062a5df157d5e166f219739f46cd05199e183c8: Status 404 returned error can't find the container with id a37d80d28951aa9f7681d6624062a5df157d5e166f219739f46cd05199e183c8 Jul 15 21:34:18.438812 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.438735 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:18.491399 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.491370 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:18.975704 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.975669 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" event={"ID":"df419a7c-0a7c-47aa-a3f8-de27031cd584","Type":"ContainerStarted","Data":"0bf80ab6516626a57150151f566b7e7615b01ca848d12d637897a161b0300569"} Jul 15 21:34:18.975704 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.975708 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" event={"ID":"df419a7c-0a7c-47aa-a3f8-de27031cd584","Type":"ContainerStarted","Data":"a37d80d28951aa9f7681d6624062a5df157d5e166f219739f46cd05199e183c8"} Jul 15 21:34:18.975924 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:18.975769 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:20.855709 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:20.855672 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:20.984391 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:20.984336 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" podStartSLOduration=3.984317431 podStartE2EDuration="3.984317431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:34:18.99082762 +0000 UTC m=+255.571424648" watchObservedRunningTime="2026-07-15 21:34:20.984317431 +0000 UTC m=+257.564914462" Jul 15 21:34:20.984823 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:20.984804 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kghn6"] Jul 15 21:34:20.987388 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:20.987366 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:20.989672 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:20.989656 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-g2p9v\"" Jul 15 21:34:20.995841 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:20.995821 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kghn6"] Jul 15 21:34:21.071794 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.071765 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cfb8186-cbba-400e-a37d-9ebb86281a5f-bound-sa-token\") pod \"cert-manager-55b4d954d5-kghn6\" (UID: \"9cfb8186-cbba-400e-a37d-9ebb86281a5f\") " pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.071959 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.071812 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xs4v\" (UniqueName: \"kubernetes.io/projected/9cfb8186-cbba-400e-a37d-9ebb86281a5f-kube-api-access-9xs4v\") pod \"cert-manager-55b4d954d5-kghn6\" (UID: \"9cfb8186-cbba-400e-a37d-9ebb86281a5f\") " pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.172639 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.172556 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cfb8186-cbba-400e-a37d-9ebb86281a5f-bound-sa-token\") pod \"cert-manager-55b4d954d5-kghn6\" (UID: \"9cfb8186-cbba-400e-a37d-9ebb86281a5f\") " pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.172639 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.172598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xs4v\" (UniqueName: \"kubernetes.io/projected/9cfb8186-cbba-400e-a37d-9ebb86281a5f-kube-api-access-9xs4v\") pod \"cert-manager-55b4d954d5-kghn6\" (UID: \"9cfb8186-cbba-400e-a37d-9ebb86281a5f\") " pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.180563 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.180531 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9cfb8186-cbba-400e-a37d-9ebb86281a5f-bound-sa-token\") pod \"cert-manager-55b4d954d5-kghn6\" (UID: \"9cfb8186-cbba-400e-a37d-9ebb86281a5f\") " pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.180676 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.180589 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xs4v\" (UniqueName: \"kubernetes.io/projected/9cfb8186-cbba-400e-a37d-9ebb86281a5f-kube-api-access-9xs4v\") pod \"cert-manager-55b4d954d5-kghn6\" (UID: \"9cfb8186-cbba-400e-a37d-9ebb86281a5f\") " pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.297289 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.297261 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kghn6" Jul 15 21:34:21.429891 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.429870 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kghn6"] Jul 15 21:34:21.432315 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:21.432288 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9cfb8186_cbba_400e_a37d_9ebb86281a5f.slice/crio-b6dc3d199dabfe7b4836ec256b86b0c5bb43e4b61500e23d83078e9c3c9a541b WatchSource:0}: Error finding container b6dc3d199dabfe7b4836ec256b86b0c5bb43e4b61500e23d83078e9c3c9a541b: Status 404 returned error can't find the container with id b6dc3d199dabfe7b4836ec256b86b0c5bb43e4b61500e23d83078e9c3c9a541b Jul 15 21:34:21.835774 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.835745 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:21.936741 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.936699 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:21.987433 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.987406 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kghn6" event={"ID":"9cfb8186-cbba-400e-a37d-9ebb86281a5f","Type":"ContainerStarted","Data":"8abc7d4678ab3e648680a1cc8d488d0fd93d12d5a0ab8abab0b078b3a9f7dedb"} Jul 15 21:34:21.987563 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:21.987440 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kghn6" event={"ID":"9cfb8186-cbba-400e-a37d-9ebb86281a5f","Type":"ContainerStarted","Data":"b6dc3d199dabfe7b4836ec256b86b0c5bb43e4b61500e23d83078e9c3c9a541b"} Jul 15 21:34:22.001802 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:22.001759 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kghn6" podStartSLOduration=2.001746677 podStartE2EDuration="2.001746677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:34:22.000942655 +0000 UTC m=+258.581539709" watchObservedRunningTime="2026-07-15 21:34:22.001746677 +0000 UTC m=+258.582343706" Jul 15 21:34:23.863072 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.863037 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8"] Jul 15 21:34:23.866093 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.866077 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:23.868396 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.868375 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 21:34:23.869328 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.869309 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 21:34:23.869415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.869395 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-w48rv\"" Jul 15 21:34:23.873618 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.873601 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8"] Jul 15 21:34:23.993018 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.992991 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2247ae8a-5566-4f43-a2ca-a3a32e7596ad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-l92b8\" (UID: \"2247ae8a-5566-4f43-a2ca-a3a32e7596ad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:23.993018 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:23.993020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s682\" (UniqueName: \"kubernetes.io/projected/2247ae8a-5566-4f43-a2ca-a3a32e7596ad-kube-api-access-6s682\") pod \"openshift-lws-operator-fd8ccff4c-l92b8\" (UID: \"2247ae8a-5566-4f43-a2ca-a3a32e7596ad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:24.093870 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.093842 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2247ae8a-5566-4f43-a2ca-a3a32e7596ad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-l92b8\" (UID: \"2247ae8a-5566-4f43-a2ca-a3a32e7596ad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:24.093870 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.093872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s682\" (UniqueName: \"kubernetes.io/projected/2247ae8a-5566-4f43-a2ca-a3a32e7596ad-kube-api-access-6s682\") pod \"openshift-lws-operator-fd8ccff4c-l92b8\" (UID: \"2247ae8a-5566-4f43-a2ca-a3a32e7596ad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:24.094291 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.094273 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2247ae8a-5566-4f43-a2ca-a3a32e7596ad-tmp\") pod \"openshift-lws-operator-fd8ccff4c-l92b8\" (UID: \"2247ae8a-5566-4f43-a2ca-a3a32e7596ad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:24.101646 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.101617 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s682\" (UniqueName: \"kubernetes.io/projected/2247ae8a-5566-4f43-a2ca-a3a32e7596ad-kube-api-access-6s682\") pod \"openshift-lws-operator-fd8ccff4c-l92b8\" (UID: \"2247ae8a-5566-4f43-a2ca-a3a32e7596ad\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:24.175468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.175405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" Jul 15 21:34:24.293050 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.293028 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8"] Jul 15 21:34:24.295300 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:24.295274 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2247ae8a_5566_4f43_a2ca_a3a32e7596ad.slice/crio-46acb5e6759fa125fd5dac64ccc5e18397075f35ce1b4138f199ee4660281b2e WatchSource:0}: Error finding container 46acb5e6759fa125fd5dac64ccc5e18397075f35ce1b4138f199ee4660281b2e: Status 404 returned error can't find the container with id 46acb5e6759fa125fd5dac64ccc5e18397075f35ce1b4138f199ee4660281b2e Jul 15 21:34:24.982277 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:24.982244 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lz2mv" Jul 15 21:34:25.006223 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:25.006192 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" event={"ID":"2247ae8a-5566-4f43-a2ca-a3a32e7596ad","Type":"ContainerStarted","Data":"46acb5e6759fa125fd5dac64ccc5e18397075f35ce1b4138f199ee4660281b2e"} Jul 15 21:34:27.015176 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:27.015129 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" event={"ID":"2247ae8a-5566-4f43-a2ca-a3a32e7596ad","Type":"ContainerStarted","Data":"7e731d78ac75c34cdfa554c138f28005a53f8ac87548ecbec2fb4f0aea909f4e"} Jul 15 21:34:27.037896 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:27.037845 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-l92b8" podStartSLOduration=2.153550248 podStartE2EDuration="4.037830053s" totalImagesPullingTime="1.884279805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:34:27.037462277 +0000 UTC m=+263.618059307" watchObservedRunningTime="2026-07-15 21:34:27.037830053 +0000 UTC m=+263.618427082" Jul 15 21:34:27.798132 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:27.798074 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:27.857385 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:27.857345 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:29.825642 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:29.825609 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:31.710974 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.710939 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg"] Jul 15 21:34:31.754646 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.754618 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg"] Jul 15 21:34:31.754797 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.754731 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.757182 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.757147 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-psgb9\"" Jul 15 21:34:31.757312 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.757202 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 15 21:34:31.758155 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.758139 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 15 21:34:31.854379 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.854349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.854563 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.854410 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.854563 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.854435 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88hqq\" (UniqueName: \"kubernetes.io/projected/22ddd298-6814-4ce0-872d-db5df0586b82-kube-api-access-88hqq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.955688 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.955648 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88hqq\" (UniqueName: \"kubernetes.io/projected/22ddd298-6814-4ce0-872d-db5df0586b82-kube-api-access-88hqq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.955889 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.955703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.955889 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.955779 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.956098 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.956079 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.956207 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.956158 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:31.963253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:31.963200 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88hqq\" (UniqueName: \"kubernetes.io/projected/22ddd298-6814-4ce0-872d-db5df0586b82-kube-api-access-88hqq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:32.063997 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:32.063969 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:32.208502 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:32.208389 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg"] Jul 15 21:34:32.211160 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:32.211122 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ddd298_6814_4ce0_872d_db5df0586b82.slice/crio-86ba1efc02982c888abcc6fb839af2ebe151575ed764ffd8eaa2fed178fc7830 WatchSource:0}: Error finding container 86ba1efc02982c888abcc6fb839af2ebe151575ed764ffd8eaa2fed178fc7830: Status 404 returned error can't find the container with id 86ba1efc02982c888abcc6fb839af2ebe151575ed764ffd8eaa2fed178fc7830 Jul 15 21:34:33.041493 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:33.041458 2572 generic.go:356] "Generic (PLEG): container finished" podID="22ddd298-6814-4ce0-872d-db5df0586b82" containerID="507ddeec6916c6825768121d039245e1775213154a47728d5b0f523fdf6eddd9" exitCode=0 Jul 15 21:34:33.041926 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:33.041534 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" event={"ID":"22ddd298-6814-4ce0-872d-db5df0586b82","Type":"ContainerDied","Data":"507ddeec6916c6825768121d039245e1775213154a47728d5b0f523fdf6eddd9"} Jul 15 21:34:33.041926 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:33.041582 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" event={"ID":"22ddd298-6814-4ce0-872d-db5df0586b82","Type":"ContainerStarted","Data":"86ba1efc02982c888abcc6fb839af2ebe151575ed764ffd8eaa2fed178fc7830"} Jul 15 21:34:35.051312 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:35.051273 2572 generic.go:356] "Generic (PLEG): container finished" podID="22ddd298-6814-4ce0-872d-db5df0586b82" containerID="03cd50a3a91c5fbb601ce79cb09fd4a2fb4c0201a1729f8e16e959dc4fba39cd" exitCode=0 Jul 15 21:34:35.051690 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:35.051354 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" event={"ID":"22ddd298-6814-4ce0-872d-db5df0586b82","Type":"ContainerDied","Data":"03cd50a3a91c5fbb601ce79cb09fd4a2fb4c0201a1729f8e16e959dc4fba39cd"} Jul 15 21:34:36.057632 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:36.057599 2572 generic.go:356] "Generic (PLEG): container finished" podID="22ddd298-6814-4ce0-872d-db5df0586b82" containerID="b2d1c860823b9901c0121bed87f036c48b2160b87dd09e14ce1fd15f2d6c469e" exitCode=0 Jul 15 21:34:36.061328 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:36.061302 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" event={"ID":"22ddd298-6814-4ce0-872d-db5df0586b82","Type":"ContainerDied","Data":"b2d1c860823b9901c0121bed87f036c48b2160b87dd09e14ce1fd15f2d6c469e"} Jul 15 21:34:37.183958 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.183931 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:37.299203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.299137 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-bundle\") pod \"22ddd298-6814-4ce0-872d-db5df0586b82\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " Jul 15 21:34:37.299203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.299215 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88hqq\" (UniqueName: \"kubernetes.io/projected/22ddd298-6814-4ce0-872d-db5df0586b82-kube-api-access-88hqq\") pod \"22ddd298-6814-4ce0-872d-db5df0586b82\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " Jul 15 21:34:37.299448 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.299247 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-util\") pod \"22ddd298-6814-4ce0-872d-db5df0586b82\" (UID: \"22ddd298-6814-4ce0-872d-db5df0586b82\") " Jul 15 21:34:37.299968 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.299938 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-bundle" (OuterVolumeSpecName: "bundle") pod "22ddd298-6814-4ce0-872d-db5df0586b82" (UID: "22ddd298-6814-4ce0-872d-db5df0586b82"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:34:37.301306 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.301276 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ddd298-6814-4ce0-872d-db5df0586b82-kube-api-access-88hqq" (OuterVolumeSpecName: "kube-api-access-88hqq") pod "22ddd298-6814-4ce0-872d-db5df0586b82" (UID: "22ddd298-6814-4ce0-872d-db5df0586b82"). InnerVolumeSpecName "kube-api-access-88hqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:34:37.305101 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.305074 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-util" (OuterVolumeSpecName: "util") pod "22ddd298-6814-4ce0-872d-db5df0586b82" (UID: "22ddd298-6814-4ce0-872d-db5df0586b82"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:34:37.400299 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.400214 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:37.400299 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.400240 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88hqq\" (UniqueName: \"kubernetes.io/projected/22ddd298-6814-4ce0-872d-db5df0586b82-kube-api-access-88hqq\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:37.400299 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:37.400250 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22ddd298-6814-4ce0-872d-db5df0586b82-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:34:38.067592 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:38.067553 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" event={"ID":"22ddd298-6814-4ce0-872d-db5df0586b82","Type":"ContainerDied","Data":"86ba1efc02982c888abcc6fb839af2ebe151575ed764ffd8eaa2fed178fc7830"} Jul 15 21:34:38.067764 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:38.067596 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afbq8qg" Jul 15 21:34:38.067764 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:38.067600 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ba1efc02982c888abcc6fb839af2ebe151575ed764ffd8eaa2fed178fc7830" Jul 15 21:34:47.727339 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:47.727302 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:47.833245 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:47.833208 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:48.334758 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.334717 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:48.535590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.535555 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:48.635822 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.635735 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:48.684151 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684120 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq"] Jul 15 21:34:48.684391 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684378 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="extract" Jul 15 21:34:48.684432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684392 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="extract" Jul 15 21:34:48.684432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684403 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="util" Jul 15 21:34:48.684432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684409 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="util" Jul 15 21:34:48.684432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684418 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="pull" Jul 15 21:34:48.684432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684424 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="pull" Jul 15 21:34:48.684576 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.684477 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ddd298-6814-4ce0-872d-db5df0586b82" containerName="extract" Jul 15 21:34:48.686842 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.686825 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.690568 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.690544 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 21:34:48.690794 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.690781 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tt66b\"" Jul 15 21:34:48.691272 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.691260 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 21:34:48.691455 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.691443 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 21:34:48.692241 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.692223 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 21:34:48.710220 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.710190 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq"] Jul 15 21:34:48.783447 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.783412 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b4d5286-535c-4397-9a32-ef2cc880529a-manifests\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.783447 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.783449 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b4d5286-535c-4397-9a32-ef2cc880529a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.783888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.783478 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b4d5286-535c-4397-9a32-ef2cc880529a-webhook-cert\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.783888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.783534 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf7lz\" (UniqueName: \"kubernetes.io/projected/2b4d5286-535c-4397-9a32-ef2cc880529a-kube-api-access-pf7lz\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.783888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.783595 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b4d5286-535c-4397-9a32-ef2cc880529a-tmp\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.842675 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.842644 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:48.884758 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.884723 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf7lz\" (UniqueName: \"kubernetes.io/projected/2b4d5286-535c-4397-9a32-ef2cc880529a-kube-api-access-pf7lz\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.884936 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.884773 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b4d5286-535c-4397-9a32-ef2cc880529a-tmp\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.884936 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.884811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b4d5286-535c-4397-9a32-ef2cc880529a-manifests\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.884936 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.884839 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b4d5286-535c-4397-9a32-ef2cc880529a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.884936 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.884882 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b4d5286-535c-4397-9a32-ef2cc880529a-webhook-cert\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.885261 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.885237 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2b4d5286-535c-4397-9a32-ef2cc880529a-manifests\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.887245 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.887188 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b4d5286-535c-4397-9a32-ef2cc880529a-tmp\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.887389 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.887365 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2b4d5286-535c-4397-9a32-ef2cc880529a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.887443 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.887378 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b4d5286-535c-4397-9a32-ef2cc880529a-webhook-cert\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.892604 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.892578 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf7lz\" (UniqueName: \"kubernetes.io/projected/2b4d5286-535c-4397-9a32-ef2cc880529a-kube-api-access-pf7lz\") pod \"opendatahub-operator-controller-manager-56fdfd95c4-9q8fq\" (UID: \"2b4d5286-535c-4397-9a32-ef2cc880529a\") " pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:48.941582 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.941550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:48.996194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:48.996152 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:49.042751 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:49.042633 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:49.119663 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:49.119628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq"] Jul 15 21:34:49.121901 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:34:49.121878 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4d5286_535c_4397_9a32_ef2cc880529a.slice/crio-b2b82a425615dd85178a7b28c3802cc2c1e039068566485c3fb71813747f46c4 WatchSource:0}: Error finding container b2b82a425615dd85178a7b28c3802cc2c1e039068566485c3fb71813747f46c4: Status 404 returned error can't find the container with id b2b82a425615dd85178a7b28c3802cc2c1e039068566485c3fb71813747f46c4 Jul 15 21:34:49.124230 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:49.124189 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:49.192759 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:49.192733 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:50.114494 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:50.114451 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" event={"ID":"2b4d5286-535c-4397-9a32-ef2cc880529a","Type":"ContainerStarted","Data":"b2b82a425615dd85178a7b28c3802cc2c1e039068566485c3fb71813747f46c4"} Jul 15 21:34:53.130410 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:53.130374 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" event={"ID":"2b4d5286-535c-4397-9a32-ef2cc880529a","Type":"ContainerStarted","Data":"ff6347480dfb5f06c2d600d59a689f2752804ab72d0916544f2ffce0c9540636"} Jul 15 21:34:54.136349 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:54.136313 2572 generic.go:356] "Generic (PLEG): container finished" podID="2b4d5286-535c-4397-9a32-ef2cc880529a" containerID="ff6347480dfb5f06c2d600d59a689f2752804ab72d0916544f2ffce0c9540636" exitCode=0 Jul 15 21:34:54.136828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:54.136360 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" event={"ID":"2b4d5286-535c-4397-9a32-ef2cc880529a","Type":"ContainerDied","Data":"ff6347480dfb5f06c2d600d59a689f2752804ab72d0916544f2ffce0c9540636"} Jul 15 21:34:55.143017 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:55.142979 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" event={"ID":"2b4d5286-535c-4397-9a32-ef2cc880529a","Type":"ContainerStarted","Data":"3988bac24af5878d70c4252322ca3a9784c7b5528fe55aa5af018e6a48e6cc50"} Jul 15 21:34:55.143442 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:55.143092 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:34:55.172771 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:55.172712 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" podStartSLOduration=2.387454765 podStartE2EDuration="7.172690103s" totalImagesPullingTime="3.725995019s" totalInitContainerRuntime="1.059240319s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:34:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:34:55.169721785 +0000 UTC m=+291.750318815" watchObservedRunningTime="2026-07-15 21:34:55.172690103 +0000 UTC m=+291.753287132" Jul 15 21:34:55.200675 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:55.200640 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:34:55.312314 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:34:55.312279 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:03.836262 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:03.836233 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:35:03.839398 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:03.839370 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:35:03.844553 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:03.844532 2572 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 21:35:06.148993 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:06.148963 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56fdfd95c4-9q8fq" Jul 15 21:35:09.279823 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.279791 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:09.400562 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.400528 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:09.665290 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.665208 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:09.746936 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.746891 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9bff-fsq5n"] Jul 15 21:35:09.750302 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.750282 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.752731 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.752713 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 21:35:09.754061 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.754033 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 21:35:09.754061 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.754047 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 21:35:09.754268 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.754069 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 21:35:09.754268 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.754047 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bfplq\"" Jul 15 21:35:09.764845 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.764823 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9bff-fsq5n"] Jul 15 21:35:09.771224 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.771202 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:09.836005 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.835982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/9ae24758-303f-40fd-a1cc-9357617871e9-kube-api-access-7l2l9\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.836121 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.836013 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae24758-303f-40fd-a1cc-9357617871e9-tmp\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.836121 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.836043 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae24758-303f-40fd-a1cc-9357617871e9-tls-certs\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.872869 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.872842 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wccdv"] Jul 15 21:35:09.875996 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.875982 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:09.878774 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.878751 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-z72hc\"" Jul 15 21:35:09.878884 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.878751 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 15 21:35:09.890493 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.890470 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wccdv"] Jul 15 21:35:09.900300 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.900271 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:09.936868 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.936812 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae24758-303f-40fd-a1cc-9357617871e9-tls-certs\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.936868 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.936864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8041204f-ce83-4e43-8321-1ce6fffc7ef1-cert\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:09.937028 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.936884 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/9ae24758-303f-40fd-a1cc-9357617871e9-kube-api-access-7l2l9\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.937028 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.936901 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cznlm\" (UniqueName: \"kubernetes.io/projected/8041204f-ce83-4e43-8321-1ce6fffc7ef1-kube-api-access-cznlm\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:09.937028 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.936930 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae24758-303f-40fd-a1cc-9357617871e9-tmp\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.939100 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.939069 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9ae24758-303f-40fd-a1cc-9357617871e9-tmp\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.939304 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.939287 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ae24758-303f-40fd-a1cc-9357617871e9-tls-certs\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:09.950226 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:09.950202 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l2l9\" (UniqueName: \"kubernetes.io/projected/9ae24758-303f-40fd-a1cc-9357617871e9-kube-api-access-7l2l9\") pod \"kube-auth-proxy-9bff-fsq5n\" (UID: \"9ae24758-303f-40fd-a1cc-9357617871e9\") " pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:10.014213 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.014180 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.037421 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.037400 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8041204f-ce83-4e43-8321-1ce6fffc7ef1-cert\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:10.037493 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.037426 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cznlm\" (UniqueName: \"kubernetes.io/projected/8041204f-ce83-4e43-8321-1ce6fffc7ef1-kube-api-access-cznlm\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:10.037540 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:35:10.037524 2572 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 15 21:35:10.037582 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:35:10.037575 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8041204f-ce83-4e43-8321-1ce6fffc7ef1-cert podName:8041204f-ce83-4e43-8321-1ce6fffc7ef1 nodeName:}" failed. No retries permitted until 2026-07-15 21:35:10.537561181 +0000 UTC m=+307.118158189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8041204f-ce83-4e43-8321-1ce6fffc7ef1-cert") pod "odh-model-controller-7998c67dc4-wccdv" (UID: "8041204f-ce83-4e43-8321-1ce6fffc7ef1") : secret "odh-model-controller-webhook-cert" not found Jul 15 21:35:10.045603 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.045577 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cznlm\" (UniqueName: \"kubernetes.io/projected/8041204f-ce83-4e43-8321-1ce6fffc7ef1-kube-api-access-cznlm\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:10.059364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.059346 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" Jul 15 21:35:10.106361 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.106321 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.190118 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.190093 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9bff-fsq5n"] Jul 15 21:35:10.192689 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:35:10.192663 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae24758_303f_40fd_a1cc_9357617871e9.slice/crio-48b05cabd181a3b61ddfb7b3794a63a6e3cf649b400834452c95b0302208e452 WatchSource:0}: Error finding container 48b05cabd181a3b61ddfb7b3794a63a6e3cf649b400834452c95b0302208e452: Status 404 returned error can't find the container with id 48b05cabd181a3b61ddfb7b3794a63a6e3cf649b400834452c95b0302208e452 Jul 15 21:35:10.194400 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.194380 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:35:10.203208 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.203160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.203670 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.203650 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" event={"ID":"9ae24758-303f-40fd-a1cc-9357617871e9","Type":"ContainerStarted","Data":"48b05cabd181a3b61ddfb7b3794a63a6e3cf649b400834452c95b0302208e452"} Jul 15 21:35:10.281534 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.281505 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.541444 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.541411 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8041204f-ce83-4e43-8321-1ce6fffc7ef1-cert\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:10.543704 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.543685 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8041204f-ce83-4e43-8321-1ce6fffc7ef1-cert\") pod \"odh-model-controller-7998c67dc4-wccdv\" (UID: \"8041204f-ce83-4e43-8321-1ce6fffc7ef1\") " pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:10.550869 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.550843 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p7dsb"] Jul 15 21:35:10.555068 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.555052 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.558072 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.558048 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 15 21:35:10.571742 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.571715 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.573656 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.573634 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p7dsb"] Jul 15 21:35:10.642530 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.642503 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271d7a1-43b6-4621-871c-b3ecb87b9817-tls-certs\") pod \"model-serving-api-54d89bd5fc-p7dsb\" (UID: \"b271d7a1-43b6-4621-871c-b3ecb87b9817\") " pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.642649 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.642540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fjp\" (UniqueName: \"kubernetes.io/projected/b271d7a1-43b6-4621-871c-b3ecb87b9817-kube-api-access-k2fjp\") pod \"model-serving-api-54d89bd5fc-p7dsb\" (UID: \"b271d7a1-43b6-4621-871c-b3ecb87b9817\") " pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.663316 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.663297 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.743781 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.743748 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271d7a1-43b6-4621-871c-b3ecb87b9817-tls-certs\") pod \"model-serving-api-54d89bd5fc-p7dsb\" (UID: \"b271d7a1-43b6-4621-871c-b3ecb87b9817\") " pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.743939 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.743792 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fjp\" (UniqueName: \"kubernetes.io/projected/b271d7a1-43b6-4621-871c-b3ecb87b9817-kube-api-access-k2fjp\") pod \"model-serving-api-54d89bd5fc-p7dsb\" (UID: \"b271d7a1-43b6-4621-871c-b3ecb87b9817\") " pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.746195 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.746151 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b271d7a1-43b6-4621-871c-b3ecb87b9817-tls-certs\") pod \"model-serving-api-54d89bd5fc-p7dsb\" (UID: \"b271d7a1-43b6-4621-871c-b3ecb87b9817\") " pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.752806 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.752780 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fjp\" (UniqueName: \"kubernetes.io/projected/b271d7a1-43b6-4621-871c-b3ecb87b9817-kube-api-access-k2fjp\") pod \"model-serving-api-54d89bd5fc-p7dsb\" (UID: \"b271d7a1-43b6-4621-871c-b3ecb87b9817\") " pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.786654 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.786628 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:10.864530 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.864485 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:10.915101 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.915066 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wccdv"] Jul 15 21:35:10.919583 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:35:10.919550 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8041204f_ce83_4e43_8321_1ce6fffc7ef1.slice/crio-7a05466c7c325d2d5ddf05e530afbf71e285379bf08b6d2ed2bfae2a85696032 WatchSource:0}: Error finding container 7a05466c7c325d2d5ddf05e530afbf71e285379bf08b6d2ed2bfae2a85696032: Status 404 returned error can't find the container with id 7a05466c7c325d2d5ddf05e530afbf71e285379bf08b6d2ed2bfae2a85696032 Jul 15 21:35:10.924350 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.924327 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz"] Jul 15 21:35:10.929786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.929764 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.934468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.934386 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b5szw\"" Jul 15 21:35:10.934468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.934397 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 15 21:35:10.934468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.934426 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 15 21:35:10.934662 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.934508 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 21:35:10.934739 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.934720 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 15 21:35:10.940529 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.940507 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz"] Jul 15 21:35:10.940640 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.940522 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:10.945655 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945630 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.945758 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945681 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.945758 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3817f01-84a5-440c-ab61-ecf69e83ca5d-local-certs\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.945841 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945757 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.945841 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945818 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq8fw\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-kube-api-access-cq8fw\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.945908 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945844 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-token\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:10.945908 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:10.945867 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-cacerts\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.016708 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.015991 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:11.018043 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.018017 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p7dsb"] Jul 15 21:35:11.046596 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046502 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.046596 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046561 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.046596 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046585 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3817f01-84a5-440c-ab61-ecf69e83ca5d-local-certs\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.046871 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.046871 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq8fw\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-kube-api-access-cq8fw\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.046871 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046665 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-token\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.046871 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.046681 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-cacerts\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.047461 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.047431 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.049483 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.049445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3817f01-84a5-440c-ab61-ecf69e83ca5d-local-certs\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.049596 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.049548 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.049657 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.049624 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.049707 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.049654 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-cacerts\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.055112 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.055090 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-token\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.055573 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.055550 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq8fw\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-kube-api-access-cq8fw\") pod \"istiod-openshift-gateway-7574459656-4vxqz\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.209931 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.209888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" event={"ID":"b271d7a1-43b6-4621-871c-b3ecb87b9817","Type":"ContainerStarted","Data":"15884fe125338627c8a520fdcfc2b82fcb3529758bf9c96f5bc2efcd342d0e61"} Jul 15 21:35:11.211104 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.211080 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" event={"ID":"8041204f-ce83-4e43-8321-1ce6fffc7ef1","Type":"ContainerStarted","Data":"7a05466c7c325d2d5ddf05e530afbf71e285379bf08b6d2ed2bfae2a85696032"} Jul 15 21:35:11.257819 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.257776 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:11.393647 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.393621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz"] Jul 15 21:35:11.395606 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:35:11.395574 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3817f01_84a5_440c_ab61_ecf69e83ca5d.slice/crio-5e52373d1d425bbdea90be3377c16e4199a93034dc3672d8b9033dbca696c9fc WatchSource:0}: Error finding container 5e52373d1d425bbdea90be3377c16e4199a93034dc3672d8b9033dbca696c9fc: Status 404 returned error can't find the container with id 5e52373d1d425bbdea90be3377c16e4199a93034dc3672d8b9033dbca696c9fc Jul 15 21:35:11.847945 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:11.847904 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:12.220598 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:12.220555 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" event={"ID":"c3817f01-84a5-440c-ab61-ecf69e83ca5d","Type":"ContainerStarted","Data":"5e52373d1d425bbdea90be3377c16e4199a93034dc3672d8b9033dbca696c9fc"} Jul 15 21:35:16.080231 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.080193 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 21:35:16.080592 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.080278 2572 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 21:35:16.242487 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.242444 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" event={"ID":"8041204f-ce83-4e43-8321-1ce6fffc7ef1","Type":"ContainerStarted","Data":"8c8344cad913cb309706fdd762212b167002a01e383bd70fe524cdcbbd733a08"} Jul 15 21:35:16.242875 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.242849 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:16.244431 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.244401 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" event={"ID":"c3817f01-84a5-440c-ab61-ecf69e83ca5d","Type":"ContainerStarted","Data":"a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94"} Jul 15 21:35:16.245000 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.244973 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:16.246575 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.246502 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" event={"ID":"9ae24758-303f-40fd-a1cc-9357617871e9","Type":"ContainerStarted","Data":"8473061b6c7ba6e74c7ea52b442bba170a47e704e398cde43d21692f4da755ae"} Jul 15 21:35:16.248301 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.248277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" event={"ID":"b271d7a1-43b6-4621-871c-b3ecb87b9817","Type":"ContainerStarted","Data":"341ec1fe369d5d4bbdf4663754ef895ee9ef4af6ca4e8b4a3e10ac53129ba669"} Jul 15 21:35:16.248710 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.248691 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:16.263123 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.263077 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" podStartSLOduration=2.105041944 podStartE2EDuration="7.263062915s" totalImagesPullingTime="5.158020971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:35:16.261536433 +0000 UTC m=+312.842133470" watchObservedRunningTime="2026-07-15 21:35:16.263062915 +0000 UTC m=+312.843659944" Jul 15 21:35:16.283184 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.283102 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" podStartSLOduration=1.237610167 podStartE2EDuration="6.283087134s" totalImagesPullingTime="5.045476967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:35:16.281273043 +0000 UTC m=+312.861870107" watchObservedRunningTime="2026-07-15 21:35:16.283087134 +0000 UTC m=+312.863684162" Jul 15 21:35:16.300373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.300321 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" podStartSLOduration=1.6181980660000002 podStartE2EDuration="6.30028569s" totalImagesPullingTime="4.682087624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:35:16.299981732 +0000 UTC m=+312.880578795" watchObservedRunningTime="2026-07-15 21:35:16.30028569 +0000 UTC m=+312.880882720" Jul 15 21:35:16.316694 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.316605 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9bff-fsq5n" podStartSLOduration=1.4466799670000001 podStartE2EDuration="7.316587593s" totalImagesPullingTime="5.869907626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:35:16.315049664 +0000 UTC m=+312.895646693" watchObservedRunningTime="2026-07-15 21:35:16.316587593 +0000 UTC m=+312.897184623" Jul 15 21:35:16.991883 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:16.991834 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:17.094252 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:17.094218 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:17.255747 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:17.255674 2572 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-4vxqz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 21:35:17.255747 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:17.255733 2572 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" podUID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 21:35:17.560094 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:17.560021 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:17.908877 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:17.908796 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:18.055945 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:18.055916 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:18.259051 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:18.259022 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:20.135274 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.135241 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:20.237185 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.237144 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:20.339492 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.339455 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4"] Jul 15 21:35:20.345318 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.344446 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.347340 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.347316 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sp7hq\"" Jul 15 21:35:20.347340 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.347328 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 21:35:20.347687 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.347670 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 21:35:20.347761 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.347689 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 21:35:20.353629 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.353604 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4"] Jul 15 21:35:20.426810 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.426729 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-manager-config\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.426810 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.426797 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-cert\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.426979 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.426855 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-metrics-cert\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.426979 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.426911 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25npl\" (UniqueName: \"kubernetes.io/projected/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-kube-api-access-25npl\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.527543 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.527506 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-manager-config\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.527695 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.527555 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-cert\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.527695 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.527577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-metrics-cert\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.527695 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.527600 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25npl\" (UniqueName: \"kubernetes.io/projected/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-kube-api-access-25npl\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.528355 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.528329 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-manager-config\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.530094 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.530074 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-cert\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.530174 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.530122 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-metrics-cert\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.549708 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.549682 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25npl\" (UniqueName: \"kubernetes.io/projected/55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2-kube-api-access-25npl\") pod \"lws-controller-manager-6577974b46-4zvd4\" (UID: \"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2\") " pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.654908 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.654873 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:20.724660 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.724628 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:20.794813 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:20.794786 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4"] Jul 15 21:35:20.797241 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:35:20.797215 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d1bf71_b2a7_4f11_8fcf_fa83e46b13c2.slice/crio-3c6de757470ed7dfb68d6e67fa6970d2bb931155217a7a693f9322523f0a41b2 WatchSource:0}: Error finding container 3c6de757470ed7dfb68d6e67fa6970d2bb931155217a7a693f9322523f0a41b2: Status 404 returned error can't find the container with id 3c6de757470ed7dfb68d6e67fa6970d2bb931155217a7a693f9322523f0a41b2 Jul 15 21:35:21.273365 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:21.273330 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" event={"ID":"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2","Type":"ContainerStarted","Data":"3c6de757470ed7dfb68d6e67fa6970d2bb931155217a7a693f9322523f0a41b2"} Jul 15 21:35:21.529240 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:21.529147 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:21.621470 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:21.621434 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:21.715620 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:21.715588 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz"] Jul 15 21:35:21.715785 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:21.715764 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" podUID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" containerName="discovery" containerID="cri-o://a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94" gracePeriod=30 Jul 15 21:35:21.959694 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:21.959665 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:22.041720 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041684 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3817f01-84a5-440c-ab61-ecf69e83ca5d-local-certs\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.041912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041732 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq8fw\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-kube-api-access-cq8fw\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.041912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041787 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-token\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.041912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041829 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-dns-cert\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.041912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041871 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-kubeconfig\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.041912 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041897 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-cacerts\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.042188 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.041947 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-ca-configmap\") pod \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\" (UID: \"c3817f01-84a5-440c-ab61-ecf69e83ca5d\") " Jul 15 21:35:22.042490 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.042459 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 21:35:22.044251 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.044147 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3817f01-84a5-440c-ab61-ecf69e83ca5d-local-certs" (OuterVolumeSpecName: "local-certs") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:35:22.044251 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.044201 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:35:22.044481 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.044457 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:35:22.044481 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.044472 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-cacerts" (OuterVolumeSpecName: "cacerts") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:35:22.044481 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.044461 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:35:22.044654 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.044562 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-kube-api-access-cq8fw" (OuterVolumeSpecName: "kube-api-access-cq8fw") pod "c3817f01-84a5-440c-ab61-ecf69e83ca5d" (UID: "c3817f01-84a5-440c-ab61-ecf69e83ca5d"). InnerVolumeSpecName "kube-api-access-cq8fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142676 2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-ca-configmap\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142705 2572 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3817f01-84a5-440c-ab61-ecf69e83ca5d-local-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142718 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq8fw\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-kube-api-access-cq8fw\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142731 2572 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-token\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142743 2572 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-csr-dns-cert\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142754 2572 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-istio-kubeconfig\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.142763 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.142765 2572 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c3817f01-84a5-440c-ab61-ecf69e83ca5d-cacerts\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:35:22.278579 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.278548 2572 generic.go:356] "Generic (PLEG): container finished" podID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" containerID="a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94" exitCode=0 Jul 15 21:35:22.278957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.278590 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" Jul 15 21:35:22.278957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.278613 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" event={"ID":"c3817f01-84a5-440c-ab61-ecf69e83ca5d","Type":"ContainerDied","Data":"a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94"} Jul 15 21:35:22.278957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.278650 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz" event={"ID":"c3817f01-84a5-440c-ab61-ecf69e83ca5d","Type":"ContainerDied","Data":"5e52373d1d425bbdea90be3377c16e4199a93034dc3672d8b9033dbca696c9fc"} Jul 15 21:35:22.278957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.278669 2572 scope.go:117] "RemoveContainer" containerID="a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94" Jul 15 21:35:22.286299 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.286282 2572 scope.go:117] "RemoveContainer" containerID="a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94" Jul 15 21:35:22.286584 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:35:22.286568 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94\": container with ID starting with a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94 not found: ID does not exist" containerID="a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94" Jul 15 21:35:22.286631 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.286594 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94"} err="failed to get container status \"a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94\": rpc error: code = NotFound desc = could not find container \"a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94\": container with ID starting with a8f8a8875206226aa2d6055e6fe8bb76ac86a3298797dbf5c6a1395680ff8a94 not found: ID does not exist" Jul 15 21:35:22.296224 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.296204 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz"] Jul 15 21:35:22.301268 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:22.301238 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-4vxqz"] Jul 15 21:35:23.285675 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:23.285640 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" event={"ID":"55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2","Type":"ContainerStarted","Data":"099f87462b14c07285aed22abbfa7910d578a9a212071070fadba4effd5be614"} Jul 15 21:35:23.286095 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:23.285700 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:23.303781 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:23.303730 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" podStartSLOduration=0.951820883 podStartE2EDuration="3.303716105s" totalImagesPullingTime="2.351895222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:35:23.302624675 +0000 UTC m=+319.883221722" watchObservedRunningTime="2026-07-15 21:35:23.303716105 +0000 UTC m=+319.884313134" Jul 15 21:35:24.062233 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:24.062196 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" path="/var/lib/kubelet/pods/c3817f01-84a5-440c-ab61-ecf69e83ca5d/volumes" Jul 15 21:35:27.255773 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:27.255740 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wccdv" Jul 15 21:35:28.264052 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:28.264024 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p7dsb" Jul 15 21:35:34.292651 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:34.292619 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6577974b46-4zvd4" Jul 15 21:35:46.483379 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:46.483340 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:57.176104 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:57.176029 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:58.500706 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:58.500679 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:58.629934 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:58.629899 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:58.690214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:58.690148 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:59.349246 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:59.349216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:35:59.449179 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:35:59.449133 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:08.996687 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:08.996646 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:09.097796 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:09.097766 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:09.620532 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:09.620498 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:10.700187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:10.700141 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:10.761295 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:10.761267 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:11.307642 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:11.307612 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:11.768521 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:11.768473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:11.854752 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:11.854719 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:26.447136 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.447100 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q"] Jul 15 21:36:26.447586 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.447400 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" containerName="discovery" Jul 15 21:36:26.447586 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.447410 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" containerName="discovery" Jul 15 21:36:26.447586 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.447461 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3817f01-84a5-440c-ab61-ecf69e83ca5d" containerName="discovery" Jul 15 21:36:26.450359 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.450343 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.452882 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.452861 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 21:36:26.453001 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.452866 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-89klg\"" Jul 15 21:36:26.453805 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.453791 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 21:36:26.463552 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.463522 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q"] Jul 15 21:36:26.521107 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.521059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.521107 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.521111 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xfnw\" (UniqueName: \"kubernetes.io/projected/47b2923f-e3ad-408a-9e45-85512fb7f5a0-kube-api-access-6xfnw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.521370 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.521260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.622253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.622208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.622253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.622258 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.622517 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.622292 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xfnw\" (UniqueName: \"kubernetes.io/projected/47b2923f-e3ad-408a-9e45-85512fb7f5a0-kube-api-access-6xfnw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.622689 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.622664 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.622747 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.622683 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.630347 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.630324 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xfnw\" (UniqueName: \"kubernetes.io/projected/47b2923f-e3ad-408a-9e45-85512fb7f5a0-kube-api-access-6xfnw\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.759697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.759606 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:26.884229 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:26.884201 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q"] Jul 15 21:36:26.886621 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:36:26.886597 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b2923f_e3ad_408a_9e45_85512fb7f5a0.slice/crio-c406f072e89b6de7e1a18e5d7d69e39e64d8e06a4f91f917c87b715164ed02fe WatchSource:0}: Error finding container c406f072e89b6de7e1a18e5d7d69e39e64d8e06a4f91f917c87b715164ed02fe: Status 404 returned error can't find the container with id c406f072e89b6de7e1a18e5d7d69e39e64d8e06a4f91f917c87b715164ed02fe Jul 15 21:36:27.243914 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.243879 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d"] Jul 15 21:36:27.247118 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.247100 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.255399 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.255377 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d"] Jul 15 21:36:27.328584 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.328545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.328584 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.328592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfxj5\" (UniqueName: \"kubernetes.io/projected/b15bb427-5c77-4b3e-9491-5a2c47c162e7-kube-api-access-vfxj5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.328817 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.328662 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.428992 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.428954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.428992 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.428998 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfxj5\" (UniqueName: \"kubernetes.io/projected/b15bb427-5c77-4b3e-9491-5a2c47c162e7-kube-api-access-vfxj5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.429276 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.429052 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.429362 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.429340 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.429437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.429416 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.437908 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.437885 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfxj5\" (UniqueName: \"kubernetes.io/projected/b15bb427-5c77-4b3e-9491-5a2c47c162e7-kube-api-access-vfxj5\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.556215 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.556132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:27.562365 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.562338 2572 generic.go:356] "Generic (PLEG): container finished" podID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerID="6e28d67421dbbb75c1760a0158817e7ab5be8b57af16d99b5003d2acae351b37" exitCode=0 Jul 15 21:36:27.562480 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.562422 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" event={"ID":"47b2923f-e3ad-408a-9e45-85512fb7f5a0","Type":"ContainerDied","Data":"6e28d67421dbbb75c1760a0158817e7ab5be8b57af16d99b5003d2acae351b37"} Jul 15 21:36:27.562480 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.562457 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" event={"ID":"47b2923f-e3ad-408a-9e45-85512fb7f5a0","Type":"ContainerStarted","Data":"c406f072e89b6de7e1a18e5d7d69e39e64d8e06a4f91f917c87b715164ed02fe"} Jul 15 21:36:27.683933 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.683848 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d"] Jul 15 21:36:27.686267 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:36:27.686242 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15bb427_5c77_4b3e_9491_5a2c47c162e7.slice/crio-51064aff21a9a7c5ae5f8b77eb754e3a03e35dc0bafe22ef8809b14f682f555a WatchSource:0}: Error finding container 51064aff21a9a7c5ae5f8b77eb754e3a03e35dc0bafe22ef8809b14f682f555a: Status 404 returned error can't find the container with id 51064aff21a9a7c5ae5f8b77eb754e3a03e35dc0bafe22ef8809b14f682f555a Jul 15 21:36:27.847322 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.847296 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs"] Jul 15 21:36:27.850363 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.850347 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:27.857601 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.857579 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs"] Jul 15 21:36:27.934279 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.934241 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9tsd\" (UniqueName: \"kubernetes.io/projected/bdbac99f-8970-48b4-90d0-d1e47726064c-kube-api-access-m9tsd\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:27.934279 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.934279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:27.934469 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:27.934367 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.035047 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.035008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9tsd\" (UniqueName: \"kubernetes.io/projected/bdbac99f-8970-48b4-90d0-d1e47726064c-kube-api-access-m9tsd\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.035047 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.035048 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.035278 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.035114 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.035491 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.035472 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.035527 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.035509 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.043272 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.043248 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9tsd\" (UniqueName: \"kubernetes.io/projected/bdbac99f-8970-48b4-90d0-d1e47726064c-kube-api-access-m9tsd\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.181777 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.181694 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:28.249600 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.249567 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt"] Jul 15 21:36:28.253201 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.253175 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.262532 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.262352 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt"] Jul 15 21:36:28.312120 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.312094 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs"] Jul 15 21:36:28.314473 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:36:28.314449 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbac99f_8970_48b4_90d0_d1e47726064c.slice/crio-bad882395d3b0f749cf917da009ecb0f818fa751798af8db0a1bc5a798d3506d WatchSource:0}: Error finding container bad882395d3b0f749cf917da009ecb0f818fa751798af8db0a1bc5a798d3506d: Status 404 returned error can't find the container with id bad882395d3b0f749cf917da009ecb0f818fa751798af8db0a1bc5a798d3506d Jul 15 21:36:28.338426 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.338398 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.338544 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.338441 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-kube-api-access-75qt7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.338544 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.338508 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.439960 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.439887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.439960 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.439929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-kube-api-access-75qt7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.440192 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.440054 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.440340 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.440321 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.440407 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.440349 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.448507 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.448485 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-kube-api-access-75qt7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.564588 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.564559 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:28.568983 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.568952 2572 generic.go:356] "Generic (PLEG): container finished" podID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerID="7061bce64f1454b609b968f062b74083dc34317f92b16ea31fdfcd6690338f51" exitCode=0 Jul 15 21:36:28.569105 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.569038 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" event={"ID":"bdbac99f-8970-48b4-90d0-d1e47726064c","Type":"ContainerDied","Data":"7061bce64f1454b609b968f062b74083dc34317f92b16ea31fdfcd6690338f51"} Jul 15 21:36:28.569105 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.569083 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" event={"ID":"bdbac99f-8970-48b4-90d0-d1e47726064c","Type":"ContainerStarted","Data":"bad882395d3b0f749cf917da009ecb0f818fa751798af8db0a1bc5a798d3506d"} Jul 15 21:36:28.570794 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.570770 2572 generic.go:356] "Generic (PLEG): container finished" podID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerID="3bed99a85046357cb1fdfccdaeba6db095bce920b7ada078c730d0c9d471c44a" exitCode=0 Jul 15 21:36:28.570909 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.570818 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" event={"ID":"b15bb427-5c77-4b3e-9491-5a2c47c162e7","Type":"ContainerDied","Data":"3bed99a85046357cb1fdfccdaeba6db095bce920b7ada078c730d0c9d471c44a"} Jul 15 21:36:28.570909 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.570853 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" event={"ID":"b15bb427-5c77-4b3e-9491-5a2c47c162e7","Type":"ContainerStarted","Data":"51064aff21a9a7c5ae5f8b77eb754e3a03e35dc0bafe22ef8809b14f682f555a"} Jul 15 21:36:28.708182 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:28.708136 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt"] Jul 15 21:36:28.710595 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:36:28.710564 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f84e7a5_99db_4d0d_a6bc_eb7cf18e2036.slice/crio-f447f5ad1d5409a688f9e04730aa3d26b4e6a4b36ef7737484fe4ea9b19a2697 WatchSource:0}: Error finding container f447f5ad1d5409a688f9e04730aa3d26b4e6a4b36ef7737484fe4ea9b19a2697: Status 404 returned error can't find the container with id f447f5ad1d5409a688f9e04730aa3d26b4e6a4b36ef7737484fe4ea9b19a2697 Jul 15 21:36:29.577485 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.577448 2572 generic.go:356] "Generic (PLEG): container finished" podID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerID="b47061ed245cd64b6d1b7c6b295f492aba04a1ff06951597ac7497fc53c8a74e" exitCode=0 Jul 15 21:36:29.577870 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.577506 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" event={"ID":"47b2923f-e3ad-408a-9e45-85512fb7f5a0","Type":"ContainerDied","Data":"b47061ed245cd64b6d1b7c6b295f492aba04a1ff06951597ac7497fc53c8a74e"} Jul 15 21:36:29.579344 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.579315 2572 generic.go:356] "Generic (PLEG): container finished" podID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerID="537a3e31b2748efa5f1d5d28d0d58f6a8656d5459bafc7f14853b573dc2da470" exitCode=0 Jul 15 21:36:29.579433 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.579344 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" event={"ID":"bdbac99f-8970-48b4-90d0-d1e47726064c","Type":"ContainerDied","Data":"537a3e31b2748efa5f1d5d28d0d58f6a8656d5459bafc7f14853b573dc2da470"} Jul 15 21:36:29.581344 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.581319 2572 generic.go:356] "Generic (PLEG): container finished" podID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerID="3647181bd94a723767039a42d4b97959edfdc4a2e211f0acb8b0c05228b4d3b2" exitCode=0 Jul 15 21:36:29.581432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.581381 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" event={"ID":"b15bb427-5c77-4b3e-9491-5a2c47c162e7","Type":"ContainerDied","Data":"3647181bd94a723767039a42d4b97959edfdc4a2e211f0acb8b0c05228b4d3b2"} Jul 15 21:36:29.582803 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.582777 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerID="78254186a6a0cb93599a379f75294ace37a2a1cd1c2998074b9eefa02aadbae6" exitCode=0 Jul 15 21:36:29.582918 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.582821 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" event={"ID":"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036","Type":"ContainerDied","Data":"78254186a6a0cb93599a379f75294ace37a2a1cd1c2998074b9eefa02aadbae6"} Jul 15 21:36:29.582918 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:29.582852 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" event={"ID":"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036","Type":"ContainerStarted","Data":"f447f5ad1d5409a688f9e04730aa3d26b4e6a4b36ef7737484fe4ea9b19a2697"} Jul 15 21:36:30.589134 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.589100 2572 generic.go:356] "Generic (PLEG): container finished" podID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerID="bcd917193772d8d2ec866e9dfd3557143cf2610e03910c654270ef1bbb1b044f" exitCode=0 Jul 15 21:36:30.589533 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.589205 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" event={"ID":"b15bb427-5c77-4b3e-9491-5a2c47c162e7","Type":"ContainerDied","Data":"bcd917193772d8d2ec866e9dfd3557143cf2610e03910c654270ef1bbb1b044f"} Jul 15 21:36:30.590827 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.590807 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerID="5c49f565becda0e34ad02eecc860e5a42b3068efaa26d73929a2341eefc2d48c" exitCode=0 Jul 15 21:36:30.590940 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.590896 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" event={"ID":"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036","Type":"ContainerDied","Data":"5c49f565becda0e34ad02eecc860e5a42b3068efaa26d73929a2341eefc2d48c"} Jul 15 21:36:30.593056 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.593035 2572 generic.go:356] "Generic (PLEG): container finished" podID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerID="03c9f5a8814bbe1d5095b9b54539ca80183be04d42fdd45f90999054c31be367" exitCode=0 Jul 15 21:36:30.593196 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.593116 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" event={"ID":"47b2923f-e3ad-408a-9e45-85512fb7f5a0","Type":"ContainerDied","Data":"03c9f5a8814bbe1d5095b9b54539ca80183be04d42fdd45f90999054c31be367"} Jul 15 21:36:30.594965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.594948 2572 generic.go:356] "Generic (PLEG): container finished" podID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerID="ffc39a023412594bed08956aececf0a3210b9bc426355ba17a4e6e183d6015d5" exitCode=0 Jul 15 21:36:30.595037 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:30.595009 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" event={"ID":"bdbac99f-8970-48b4-90d0-d1e47726064c","Type":"ContainerDied","Data":"ffc39a023412594bed08956aececf0a3210b9bc426355ba17a4e6e183d6015d5"} Jul 15 21:36:31.601759 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.601724 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerID="768f7db1354c131a67585072f9aabe14755de89f576b945dd70a5a06f458d8e6" exitCode=0 Jul 15 21:36:31.602160 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.601824 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" event={"ID":"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036","Type":"ContainerDied","Data":"768f7db1354c131a67585072f9aabe14755de89f576b945dd70a5a06f458d8e6"} Jul 15 21:36:31.765388 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.765367 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:31.768523 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.768504 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:31.792935 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.792913 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:31.868665 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868580 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xfnw\" (UniqueName: \"kubernetes.io/projected/47b2923f-e3ad-408a-9e45-85512fb7f5a0-kube-api-access-6xfnw\") pod \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " Jul 15 21:36:31.868665 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868627 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-util\") pod \"bdbac99f-8970-48b4-90d0-d1e47726064c\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " Jul 15 21:36:31.868665 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868646 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-util\") pod \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " Jul 15 21:36:31.868665 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868666 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-bundle\") pod \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\" (UID: \"47b2923f-e3ad-408a-9e45-85512fb7f5a0\") " Jul 15 21:36:31.869037 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868710 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-bundle\") pod \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " Jul 15 21:36:31.869037 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868727 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-util\") pod \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " Jul 15 21:36:31.869037 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868749 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-bundle\") pod \"bdbac99f-8970-48b4-90d0-d1e47726064c\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " Jul 15 21:36:31.869037 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868807 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9tsd\" (UniqueName: \"kubernetes.io/projected/bdbac99f-8970-48b4-90d0-d1e47726064c-kube-api-access-m9tsd\") pod \"bdbac99f-8970-48b4-90d0-d1e47726064c\" (UID: \"bdbac99f-8970-48b4-90d0-d1e47726064c\") " Jul 15 21:36:31.869037 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.868837 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfxj5\" (UniqueName: \"kubernetes.io/projected/b15bb427-5c77-4b3e-9491-5a2c47c162e7-kube-api-access-vfxj5\") pod \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\" (UID: \"b15bb427-5c77-4b3e-9491-5a2c47c162e7\") " Jul 15 21:36:31.869416 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.869389 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-bundle" (OuterVolumeSpecName: "bundle") pod "47b2923f-e3ad-408a-9e45-85512fb7f5a0" (UID: "47b2923f-e3ad-408a-9e45-85512fb7f5a0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:31.869516 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.869410 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-bundle" (OuterVolumeSpecName: "bundle") pod "b15bb427-5c77-4b3e-9491-5a2c47c162e7" (UID: "b15bb427-5c77-4b3e-9491-5a2c47c162e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:31.870072 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.870037 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-bundle" (OuterVolumeSpecName: "bundle") pod "bdbac99f-8970-48b4-90d0-d1e47726064c" (UID: "bdbac99f-8970-48b4-90d0-d1e47726064c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:31.871185 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.871146 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15bb427-5c77-4b3e-9491-5a2c47c162e7-kube-api-access-vfxj5" (OuterVolumeSpecName: "kube-api-access-vfxj5") pod "b15bb427-5c77-4b3e-9491-5a2c47c162e7" (UID: "b15bb427-5c77-4b3e-9491-5a2c47c162e7"). InnerVolumeSpecName "kube-api-access-vfxj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:36:31.871394 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.871361 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b2923f-e3ad-408a-9e45-85512fb7f5a0-kube-api-access-6xfnw" (OuterVolumeSpecName: "kube-api-access-6xfnw") pod "47b2923f-e3ad-408a-9e45-85512fb7f5a0" (UID: "47b2923f-e3ad-408a-9e45-85512fb7f5a0"). InnerVolumeSpecName "kube-api-access-6xfnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:36:31.871795 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.871766 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbac99f-8970-48b4-90d0-d1e47726064c-kube-api-access-m9tsd" (OuterVolumeSpecName: "kube-api-access-m9tsd") pod "bdbac99f-8970-48b4-90d0-d1e47726064c" (UID: "bdbac99f-8970-48b4-90d0-d1e47726064c"). InnerVolumeSpecName "kube-api-access-m9tsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:36:31.874907 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.874879 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-util" (OuterVolumeSpecName: "util") pod "b15bb427-5c77-4b3e-9491-5a2c47c162e7" (UID: "b15bb427-5c77-4b3e-9491-5a2c47c162e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:31.875555 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.875535 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-util" (OuterVolumeSpecName: "util") pod "47b2923f-e3ad-408a-9e45-85512fb7f5a0" (UID: "47b2923f-e3ad-408a-9e45-85512fb7f5a0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:31.875622 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.875596 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-util" (OuterVolumeSpecName: "util") pod "bdbac99f-8970-48b4-90d0-d1e47726064c" (UID: "bdbac99f-8970-48b4-90d0-d1e47726064c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:31.970362 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970325 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970362 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970355 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/47b2923f-e3ad-408a-9e45-85512fb7f5a0-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970362 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970364 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970362 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970372 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b15bb427-5c77-4b3e-9491-5a2c47c162e7-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970380 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970390 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9tsd\" (UniqueName: \"kubernetes.io/projected/bdbac99f-8970-48b4-90d0-d1e47726064c-kube-api-access-m9tsd\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970399 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfxj5\" (UniqueName: \"kubernetes.io/projected/b15bb427-5c77-4b3e-9491-5a2c47c162e7-kube-api-access-vfxj5\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970409 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xfnw\" (UniqueName: \"kubernetes.io/projected/47b2923f-e3ad-408a-9e45-85512fb7f5a0-kube-api-access-6xfnw\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:31.970612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:31.970417 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bdbac99f-8970-48b4-90d0-d1e47726064c-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:32.608070 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.608031 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" event={"ID":"47b2923f-e3ad-408a-9e45-85512fb7f5a0","Type":"ContainerDied","Data":"c406f072e89b6de7e1a18e5d7d69e39e64d8e06a4f91f917c87b715164ed02fe"} Jul 15 21:36:32.608564 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.608111 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c406f072e89b6de7e1a18e5d7d69e39e64d8e06a4f91f917c87b715164ed02fe" Jul 15 21:36:32.608564 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.608043 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q" Jul 15 21:36:32.610110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.610081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" event={"ID":"bdbac99f-8970-48b4-90d0-d1e47726064c","Type":"ContainerDied","Data":"bad882395d3b0f749cf917da009ecb0f818fa751798af8db0a1bc5a798d3506d"} Jul 15 21:36:32.610253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.610144 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad882395d3b0f749cf917da009ecb0f818fa751798af8db0a1bc5a798d3506d" Jul 15 21:36:32.610253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.610095 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs" Jul 15 21:36:32.612263 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.612239 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" Jul 15 21:36:32.612351 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.612266 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d" event={"ID":"b15bb427-5c77-4b3e-9491-5a2c47c162e7","Type":"ContainerDied","Data":"51064aff21a9a7c5ae5f8b77eb754e3a03e35dc0bafe22ef8809b14f682f555a"} Jul 15 21:36:32.612351 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.612305 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51064aff21a9a7c5ae5f8b77eb754e3a03e35dc0bafe22ef8809b14f682f555a" Jul 15 21:36:32.731885 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.731847 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:32.777887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.777857 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-util\") pod \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " Jul 15 21:36:32.778040 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.777895 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-kube-api-access-75qt7\") pod \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " Jul 15 21:36:32.778040 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.777952 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-bundle\") pod \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\" (UID: \"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036\") " Jul 15 21:36:32.778591 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.778550 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-bundle" (OuterVolumeSpecName: "bundle") pod "3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" (UID: "3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:32.779982 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.779955 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-kube-api-access-75qt7" (OuterVolumeSpecName: "kube-api-access-75qt7") pod "3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" (UID: "3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036"). InnerVolumeSpecName "kube-api-access-75qt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:36:32.785713 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.785688 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-util" (OuterVolumeSpecName: "util") pod "3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" (UID: "3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 21:36:32.879486 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.879402 2572 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-util\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:32.879486 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.879431 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75qt7\" (UniqueName: \"kubernetes.io/projected/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-kube-api-access-75qt7\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:32.879486 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:32.879443 2572 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036-bundle\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:36:33.618455 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:33.618423 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" event={"ID":"3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036","Type":"ContainerDied","Data":"f447f5ad1d5409a688f9e04730aa3d26b4e6a4b36ef7737484fe4ea9b19a2697"} Jul 15 21:36:33.618827 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:33.618462 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt" Jul 15 21:36:33.618827 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:33.618469 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f447f5ad1d5409a688f9e04730aa3d26b4e6a4b36ef7737484fe4ea9b19a2697" Jul 15 21:36:40.204248 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:40.204207 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:44.384764 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:44.384728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:46.782710 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:46.782682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:47.093443 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:47.093334 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:47.396017 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:47.395941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:50.825769 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:50.825734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:51.826744 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:51.826707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:51.928004 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:51.927974 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:53.831222 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:53.831190 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:55.617578 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:55.617550 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:57.001944 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:57.001913 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:36:57.098679 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:36:57.098651 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:00.780890 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:00.780861 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:00.884857 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:00.884822 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:03.523345 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523315 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-99drq"] Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523613 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523624 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523637 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="pull" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523644 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="pull" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523653 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="util" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523659 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="util" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523664 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523669 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523675 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523680 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523685 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523689 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="extract" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523697 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="util" Jul 15 21:37:03.523697 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523702 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="util" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523707 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="pull" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523712 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="pull" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523719 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="util" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523725 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="util" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523733 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="pull" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523738 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="pull" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523744 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="util" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523748 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="util" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523758 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="pull" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523762 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="pull" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523803 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036" containerName="extract" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523811 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdbac99f-8970-48b4-90d0-d1e47726064c" containerName="extract" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523818 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b2923f-e3ad-408a-9e45-85512fb7f5a0" containerName="extract" Jul 15 21:37:03.524110 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.523824 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b15bb427-5c77-4b3e-9491-5a2c47c162e7" containerName="extract" Jul 15 21:37:03.526650 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.526635 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.529276 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.529250 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 21:37:03.529389 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.529288 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 21:37:03.530239 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.530219 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 15 21:37:03.530239 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.530236 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 15 21:37:03.530394 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.530239 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-89klg\"" Jul 15 21:37:03.535578 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.535557 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-99drq"] Jul 15 21:37:03.535771 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.535754 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:03.613187 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.613138 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kd4z\" (UniqueName: \"kubernetes.io/projected/c8680ddf-2d96-43b5-965a-b308bb1335f9-kube-api-access-5kd4z\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.613361 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.613218 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8680ddf-2d96-43b5-965a-b308bb1335f9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.613361 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.613244 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8680ddf-2d96-43b5-965a-b308bb1335f9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.714551 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.714519 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kd4z\" (UniqueName: \"kubernetes.io/projected/c8680ddf-2d96-43b5-965a-b308bb1335f9-kube-api-access-5kd4z\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.714700 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.714571 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8680ddf-2d96-43b5-965a-b308bb1335f9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.714700 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.714605 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8680ddf-2d96-43b5-965a-b308bb1335f9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.714793 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:37:03.714782 2572 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 15 21:37:03.714868 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:37:03.714857 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8680ddf-2d96-43b5-965a-b308bb1335f9-plugin-serving-cert podName:c8680ddf-2d96-43b5-965a-b308bb1335f9 nodeName:}" failed. No retries permitted until 2026-07-15 21:37:04.214837386 +0000 UTC m=+420.795434392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/c8680ddf-2d96-43b5-965a-b308bb1335f9-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-99drq" (UID: "c8680ddf-2d96-43b5-965a-b308bb1335f9") : secret "plugin-serving-cert" not found Jul 15 21:37:03.715183 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.715144 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c8680ddf-2d96-43b5-965a-b308bb1335f9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.722678 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.722654 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kd4z\" (UniqueName: \"kubernetes.io/projected/c8680ddf-2d96-43b5-965a-b308bb1335f9-kube-api-access-5kd4z\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:03.936514 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:03.936440 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:04.218588 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.218552 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8680ddf-2d96-43b5-965a-b308bb1335f9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:04.220990 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.220964 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c8680ddf-2d96-43b5-965a-b308bb1335f9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-99drq\" (UID: \"c8680ddf-2d96-43b5-965a-b308bb1335f9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:04.438286 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.438254 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-89klg\"" Jul 15 21:37:04.446555 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.446534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" Jul 15 21:37:04.572535 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.572510 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-99drq"] Jul 15 21:37:04.574455 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:37:04.574429 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8680ddf_2d96_43b5_965a_b308bb1335f9.slice/crio-3649f6e16c21484c4744c368a2bf1e647f05e7cf9e20c210d3028563d8e88ec3 WatchSource:0}: Error finding container 3649f6e16c21484c4744c368a2bf1e647f05e7cf9e20c210d3028563d8e88ec3: Status 404 returned error can't find the container with id 3649f6e16c21484c4744c368a2bf1e647f05e7cf9e20c210d3028563d8e88ec3 Jul 15 21:37:04.766910 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.766820 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" event={"ID":"c8680ddf-2d96-43b5-965a-b308bb1335f9","Type":"ContainerStarted","Data":"3649f6e16c21484c4744c368a2bf1e647f05e7cf9e20c210d3028563d8e88ec3"} Jul 15 21:37:04.979784 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:04.979750 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:05.080009 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:05.079926 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:05.407091 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:05.407006 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:05.518967 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:05.518931 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:09.170029 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:09.169994 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:09.270493 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:09.270447 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:15.839855 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:15.839810 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:15.933204 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:15.933154 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:18.652503 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:18.652469 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:18.752469 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:18.752430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:23.460437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:23.460396 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:23.562749 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:23.562699 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:26.993679 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:26.993598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:27.093820 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:27.093789 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:27.899612 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:27.899574 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" event={"ID":"c8680ddf-2d96-43b5-965a-b308bb1335f9","Type":"ContainerStarted","Data":"bea5e75a4e96f4ce8b2269f475b1de97f99176640c459204f31cb709d2ee3ca3"} Jul 15 21:37:27.919679 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:27.919628 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-99drq" podStartSLOduration=2.222185308 podStartE2EDuration="24.919612398s" totalImagesPullingTime="22.69742709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:37:27.918485728 +0000 UTC m=+444.499082758" watchObservedRunningTime="2026-07-15 21:37:27.919612398 +0000 UTC m=+444.500209427" Jul 15 21:37:31.896699 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:31.896649 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:31.958194 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:31.958140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:33.622787 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:33.622742 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:33.723256 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:33.723216 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:37.012274 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:37.012240 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:37.399515 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:37.399398 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:41.420328 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:41.420271 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:41.470994 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:41.470966 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:42.417959 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:42.417925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:42.474257 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:42.474229 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:42.691273 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:42.691247 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:43.307560 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:43.307530 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:46.819269 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:46.819234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:46.919743 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:46.919710 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.794600 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:53.794565 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.890266 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:53.890234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:53.991251 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:53.991223 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:54.157267 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.157194 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:37:54.600092 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.600060 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:54.669883 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.669830 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:37:54.755656 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.755628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:37:54.755656 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.755664 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:37:54.755907 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.755683 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-9f95h"] Jul 15 21:37:54.755907 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.755776 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:54.758198 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.758155 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 21:37:54.772945 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.772928 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9f95h"] Jul 15 21:37:54.773036 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.772952 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-4k6xv"] Jul 15 21:37:54.773076 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.773050 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:37:54.775364 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.775336 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z8cx5\"" Jul 15 21:37:54.781060 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.781040 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4k6xv"] Jul 15 21:37:54.781137 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.781126 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:37:54.799970 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.799946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr2qq\" (UniqueName: \"kubernetes.io/projected/f33899f1-4f3f-4b94-b55c-d98fe1bd9554-kube-api-access-vr2qq\") pod \"limitador-limitador-54d6d988d5-6jhcv\" (UID: \"f33899f1-4f3f-4b94-b55c-d98fe1bd9554\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:54.800253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.799976 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwf5\" (UniqueName: \"kubernetes.io/projected/7b7d0580-f225-4099-a3e3-40d98bbc262e-kube-api-access-rnwf5\") pod \"authorino-58cd955675-9f95h\" (UID: \"7b7d0580-f225-4099-a3e3-40d98bbc262e\") " pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:37:54.800253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.799995 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnf6f\" (UniqueName: \"kubernetes.io/projected/d1c4e633-3129-4e1f-a074-48604ae7edb6-kube-api-access-qnf6f\") pod \"authorino-75d556bd77-4k6xv\" (UID: \"d1c4e633-3129-4e1f-a074-48604ae7edb6\") " pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:37:54.800253 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.800079 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f33899f1-4f3f-4b94-b55c-d98fe1bd9554-config-file\") pod \"limitador-limitador-54d6d988d5-6jhcv\" (UID: \"f33899f1-4f3f-4b94-b55c-d98fe1bd9554\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:54.900935 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.900870 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr2qq\" (UniqueName: \"kubernetes.io/projected/f33899f1-4f3f-4b94-b55c-d98fe1bd9554-kube-api-access-vr2qq\") pod \"limitador-limitador-54d6d988d5-6jhcv\" (UID: \"f33899f1-4f3f-4b94-b55c-d98fe1bd9554\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:54.900935 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.900904 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwf5\" (UniqueName: \"kubernetes.io/projected/7b7d0580-f225-4099-a3e3-40d98bbc262e-kube-api-access-rnwf5\") pod \"authorino-58cd955675-9f95h\" (UID: \"7b7d0580-f225-4099-a3e3-40d98bbc262e\") " pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:37:54.900935 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.900925 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnf6f\" (UniqueName: \"kubernetes.io/projected/d1c4e633-3129-4e1f-a074-48604ae7edb6-kube-api-access-qnf6f\") pod \"authorino-75d556bd77-4k6xv\" (UID: \"d1c4e633-3129-4e1f-a074-48604ae7edb6\") " pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:37:54.901142 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.900972 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f33899f1-4f3f-4b94-b55c-d98fe1bd9554-config-file\") pod \"limitador-limitador-54d6d988d5-6jhcv\" (UID: \"f33899f1-4f3f-4b94-b55c-d98fe1bd9554\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:54.901606 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.901587 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f33899f1-4f3f-4b94-b55c-d98fe1bd9554-config-file\") pod \"limitador-limitador-54d6d988d5-6jhcv\" (UID: \"f33899f1-4f3f-4b94-b55c-d98fe1bd9554\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:54.909433 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.909413 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwf5\" (UniqueName: \"kubernetes.io/projected/7b7d0580-f225-4099-a3e3-40d98bbc262e-kube-api-access-rnwf5\") pod \"authorino-58cd955675-9f95h\" (UID: \"7b7d0580-f225-4099-a3e3-40d98bbc262e\") " pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:37:54.909518 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.909468 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnf6f\" (UniqueName: \"kubernetes.io/projected/d1c4e633-3129-4e1f-a074-48604ae7edb6-kube-api-access-qnf6f\") pod \"authorino-75d556bd77-4k6xv\" (UID: \"d1c4e633-3129-4e1f-a074-48604ae7edb6\") " pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:37:54.909830 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:54.909810 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr2qq\" (UniqueName: \"kubernetes.io/projected/f33899f1-4f3f-4b94-b55c-d98fe1bd9554-kube-api-access-vr2qq\") pod \"limitador-limitador-54d6d988d5-6jhcv\" (UID: \"f33899f1-4f3f-4b94-b55c-d98fe1bd9554\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:55.065864 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:55.065832 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:37:55.081438 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:55.081416 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:37:55.089579 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:55.089560 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:37:55.244261 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:55.244230 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:37:55.246652 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:37:55.246624 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33899f1_4f3f_4b94_b55c_d98fe1bd9554.slice/crio-d62646f69da359d9c230be32462724c2807bcb8af1221b6017cf36fc967801d0 WatchSource:0}: Error finding container d62646f69da359d9c230be32462724c2807bcb8af1221b6017cf36fc967801d0: Status 404 returned error can't find the container with id d62646f69da359d9c230be32462724c2807bcb8af1221b6017cf36fc967801d0 Jul 15 21:37:55.261309 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:55.261283 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-9f95h"] Jul 15 21:37:55.261887 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:37:55.261856 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b7d0580_f225_4099_a3e3_40d98bbc262e.slice/crio-bfcbfedd9bf54619ff1cbc936064807415a1a251dfc117711f83f0eacc7c2211 WatchSource:0}: Error finding container bfcbfedd9bf54619ff1cbc936064807415a1a251dfc117711f83f0eacc7c2211: Status 404 returned error can't find the container with id bfcbfedd9bf54619ff1cbc936064807415a1a251dfc117711f83f0eacc7c2211 Jul 15 21:37:55.285458 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:55.285440 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4k6xv"] Jul 15 21:37:55.287080 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:37:55.287051 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1c4e633_3129_4e1f_a074_48604ae7edb6.slice/crio-b8f7eec0de93380ad61ebef1f5501e6ad62cb32312dd8b7ebc5892d6753ca33a WatchSource:0}: Error finding container b8f7eec0de93380ad61ebef1f5501e6ad62cb32312dd8b7ebc5892d6753ca33a: Status 404 returned error can't find the container with id b8f7eec0de93380ad61ebef1f5501e6ad62cb32312dd8b7ebc5892d6753ca33a Jul 15 21:37:56.033507 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:56.033448 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9f95h" event={"ID":"7b7d0580-f225-4099-a3e3-40d98bbc262e","Type":"ContainerStarted","Data":"bfcbfedd9bf54619ff1cbc936064807415a1a251dfc117711f83f0eacc7c2211"} Jul 15 21:37:56.035051 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:56.035022 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" event={"ID":"f33899f1-4f3f-4b94-b55c-d98fe1bd9554","Type":"ContainerStarted","Data":"d62646f69da359d9c230be32462724c2807bcb8af1221b6017cf36fc967801d0"} Jul 15 21:37:56.036308 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:37:56.036277 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4k6xv" event={"ID":"d1c4e633-3129-4e1f-a074-48604ae7edb6","Type":"ContainerStarted","Data":"b8f7eec0de93380ad61ebef1f5501e6ad62cb32312dd8b7ebc5892d6753ca33a"} Jul 15 21:38:01.071786 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.071752 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" event={"ID":"f33899f1-4f3f-4b94-b55c-d98fe1bd9554","Type":"ContainerStarted","Data":"c11d0ee15e06da5a7eb7d4af484b9f49f93db6aa1ded69e5439806901d2c6f5f"} Jul 15 21:38:01.072263 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.071849 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:38:01.073247 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.073223 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4k6xv" event={"ID":"d1c4e633-3129-4e1f-a074-48604ae7edb6","Type":"ContainerStarted","Data":"03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b"} Jul 15 21:38:01.074538 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.074514 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9f95h" event={"ID":"7b7d0580-f225-4099-a3e3-40d98bbc262e","Type":"ContainerStarted","Data":"2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc"} Jul 15 21:38:01.086945 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.086907 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" podStartSLOduration=1.693004054 podStartE2EDuration="7.086895495s" totalImagesPullingTime="5.393891441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:38:01.085962076 +0000 UTC m=+477.666559107" watchObservedRunningTime="2026-07-15 21:38:01.086895495 +0000 UTC m=+477.667492524" Jul 15 21:38:01.099500 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.099464 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-4k6xv" podStartSLOduration=1.752501707 podStartE2EDuration="7.099451466s" totalImagesPullingTime="5.346949759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:38:01.098377034 +0000 UTC m=+477.678974063" watchObservedRunningTime="2026-07-15 21:38:01.099451466 +0000 UTC m=+477.680048496" Jul 15 21:38:01.113059 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.113007 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-9f95h" podStartSLOduration=1.740909307 podStartE2EDuration="7.112988087s" totalImagesPullingTime="5.37207878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:38:01.110923953 +0000 UTC m=+477.691520983" watchObservedRunningTime="2026-07-15 21:38:01.112988087 +0000 UTC m=+477.693585119" Jul 15 21:38:01.132680 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:01.132649 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9f95h"] Jul 15 21:38:03.083848 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:03.083801 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-9f95h" podUID="7b7d0580-f225-4099-a3e3-40d98bbc262e" containerName="authorino" containerID="cri-o://2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc" gracePeriod=30 Jul 15 21:38:03.323328 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:03.323302 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:38:03.382760 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:03.382688 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnwf5\" (UniqueName: \"kubernetes.io/projected/7b7d0580-f225-4099-a3e3-40d98bbc262e-kube-api-access-rnwf5\") pod \"7b7d0580-f225-4099-a3e3-40d98bbc262e\" (UID: \"7b7d0580-f225-4099-a3e3-40d98bbc262e\") " Jul 15 21:38:03.384694 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:03.384668 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7d0580-f225-4099-a3e3-40d98bbc262e-kube-api-access-rnwf5" (OuterVolumeSpecName: "kube-api-access-rnwf5") pod "7b7d0580-f225-4099-a3e3-40d98bbc262e" (UID: "7b7d0580-f225-4099-a3e3-40d98bbc262e"). InnerVolumeSpecName "kube-api-access-rnwf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:38:03.484092 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:03.484064 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwf5\" (UniqueName: \"kubernetes.io/projected/7b7d0580-f225-4099-a3e3-40d98bbc262e-kube-api-access-rnwf5\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:38:04.094884 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.094849 2572 generic.go:356] "Generic (PLEG): container finished" podID="7b7d0580-f225-4099-a3e3-40d98bbc262e" containerID="2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc" exitCode=0 Jul 15 21:38:04.095270 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.094887 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9f95h" event={"ID":"7b7d0580-f225-4099-a3e3-40d98bbc262e","Type":"ContainerDied","Data":"2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc"} Jul 15 21:38:04.095270 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.094896 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-9f95h" Jul 15 21:38:04.095270 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.094911 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-9f95h" event={"ID":"7b7d0580-f225-4099-a3e3-40d98bbc262e","Type":"ContainerDied","Data":"bfcbfedd9bf54619ff1cbc936064807415a1a251dfc117711f83f0eacc7c2211"} Jul 15 21:38:04.095270 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.094926 2572 scope.go:117] "RemoveContainer" containerID="2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc" Jul 15 21:38:04.103042 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.103025 2572 scope.go:117] "RemoveContainer" containerID="2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc" Jul 15 21:38:04.103295 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:38:04.103278 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc\": container with ID starting with 2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc not found: ID does not exist" containerID="2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc" Jul 15 21:38:04.103347 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.103302 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc"} err="failed to get container status \"2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc\": rpc error: code = NotFound desc = could not find container \"2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc\": container with ID starting with 2c9bab1c7b16cc1af428216bd83e70fbe9ec6a3bb967574c853268491ec0a4fc not found: ID does not exist" Jul 15 21:38:04.111152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.111130 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-9f95h"] Jul 15 21:38:04.114518 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:04.114498 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-9f95h"] Jul 15 21:38:06.061730 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:06.061691 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b7d0580-f225-4099-a3e3-40d98bbc262e" path="/var/lib/kubelet/pods/7b7d0580-f225-4099-a3e3-40d98bbc262e/volumes" Jul 15 21:38:11.969181 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:11.969125 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:12.080653 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:12.080628 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-6jhcv" Jul 15 21:38:13.767883 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:13.767847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:13.866439 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:13.866407 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:14.253183 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:14.253137 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:26.367795 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:26.367758 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:26.416119 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:26.416087 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:26.896745 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:26.896713 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:27.069706 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.069680 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:27.324873 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.324844 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-6jxpp"] Jul 15 21:38:27.325179 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.325155 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b7d0580-f225-4099-a3e3-40d98bbc262e" containerName="authorino" Jul 15 21:38:27.325179 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.325178 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b7d0580-f225-4099-a3e3-40d98bbc262e" containerName="authorino" Jul 15 21:38:27.325257 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.325231 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b7d0580-f225-4099-a3e3-40d98bbc262e" containerName="authorino" Jul 15 21:38:27.328057 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.328042 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.330341 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.330318 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 21:38:27.335042 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.335017 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-6jxpp"] Jul 15 21:38:27.372180 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.372143 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-6jxpp"] Jul 15 21:38:27.372460 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:38:27.372346 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-x4tqs tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-x4tqs tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-6jxpp" podUID="351d3a96-42eb-4959-93a6-38fd4eb2027b" Jul 15 21:38:27.397303 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.397275 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f6c9c766f-47tsj"] Jul 15 21:38:27.400305 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.400285 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.407640 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.407621 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f6c9c766f-47tsj"] Jul 15 21:38:27.472373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.472349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4tqs\" (UniqueName: \"kubernetes.io/projected/351d3a96-42eb-4959-93a6-38fd4eb2027b-kube-api-access-x4tqs\") pod \"authorino-76877994-6jxpp\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.472489 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.472384 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/351d3a96-42eb-4959-93a6-38fd4eb2027b-tls-cert\") pod \"authorino-76877994-6jxpp\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.573628 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.573594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfcm\" (UniqueName: \"kubernetes.io/projected/2116e0eb-c2a8-40d8-83b8-79b5873000d6-kube-api-access-hdfcm\") pod \"authorino-5f6c9c766f-47tsj\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.573755 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.573656 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4tqs\" (UniqueName: \"kubernetes.io/projected/351d3a96-42eb-4959-93a6-38fd4eb2027b-kube-api-access-x4tqs\") pod \"authorino-76877994-6jxpp\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.573755 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.573699 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/351d3a96-42eb-4959-93a6-38fd4eb2027b-tls-cert\") pod \"authorino-76877994-6jxpp\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.573755 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.573736 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116e0eb-c2a8-40d8-83b8-79b5873000d6-tls-cert\") pod \"authorino-5f6c9c766f-47tsj\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.576152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.576110 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/351d3a96-42eb-4959-93a6-38fd4eb2027b-tls-cert\") pod \"authorino-76877994-6jxpp\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.581005 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.580990 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4tqs\" (UniqueName: \"kubernetes.io/projected/351d3a96-42eb-4959-93a6-38fd4eb2027b-kube-api-access-x4tqs\") pod \"authorino-76877994-6jxpp\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:27.674382 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.674355 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116e0eb-c2a8-40d8-83b8-79b5873000d6-tls-cert\") pod \"authorino-5f6c9c766f-47tsj\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.674540 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.674414 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfcm\" (UniqueName: \"kubernetes.io/projected/2116e0eb-c2a8-40d8-83b8-79b5873000d6-kube-api-access-hdfcm\") pod \"authorino-5f6c9c766f-47tsj\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.676623 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.676601 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116e0eb-c2a8-40d8-83b8-79b5873000d6-tls-cert\") pod \"authorino-5f6c9c766f-47tsj\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.681368 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.681351 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfcm\" (UniqueName: \"kubernetes.io/projected/2116e0eb-c2a8-40d8-83b8-79b5873000d6-kube-api-access-hdfcm\") pod \"authorino-5f6c9c766f-47tsj\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.709342 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.709321 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:38:27.834232 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:27.834200 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f6c9c766f-47tsj"] Jul 15 21:38:28.208188 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.208134 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" event={"ID":"2116e0eb-c2a8-40d8-83b8-79b5873000d6","Type":"ContainerStarted","Data":"4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d"} Jul 15 21:38:28.208324 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.208202 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:28.208324 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.208207 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" event={"ID":"2116e0eb-c2a8-40d8-83b8-79b5873000d6","Type":"ContainerStarted","Data":"6ef7b38cbb80d2cf4644cf859249e6c7c625e00daf9af9f121f45e4a5b5ed618"} Jul 15 21:38:28.215373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.215353 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:28.222619 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.222581 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" podStartSLOduration=0.949666855 podStartE2EDuration="1.222567153s" totalImagesPullingTime="272.900298ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:38:28.221418483 +0000 UTC m=+504.802015522" watchObservedRunningTime="2026-07-15 21:38:28.222567153 +0000 UTC m=+504.803164182" Jul 15 21:38:28.246377 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.246346 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4k6xv"] Jul 15 21:38:28.246555 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.246516 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-4k6xv" podUID="d1c4e633-3129-4e1f-a074-48604ae7edb6" containerName="authorino" containerID="cri-o://03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b" gracePeriod=30 Jul 15 21:38:28.379464 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.379437 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4tqs\" (UniqueName: \"kubernetes.io/projected/351d3a96-42eb-4959-93a6-38fd4eb2027b-kube-api-access-x4tqs\") pod \"351d3a96-42eb-4959-93a6-38fd4eb2027b\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " Jul 15 21:38:28.379787 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.379481 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/351d3a96-42eb-4959-93a6-38fd4eb2027b-tls-cert\") pod \"351d3a96-42eb-4959-93a6-38fd4eb2027b\" (UID: \"351d3a96-42eb-4959-93a6-38fd4eb2027b\") " Jul 15 21:38:28.381472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.381442 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/351d3a96-42eb-4959-93a6-38fd4eb2027b-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "351d3a96-42eb-4959-93a6-38fd4eb2027b" (UID: "351d3a96-42eb-4959-93a6-38fd4eb2027b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:38:28.381472 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.381443 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351d3a96-42eb-4959-93a6-38fd4eb2027b-kube-api-access-x4tqs" (OuterVolumeSpecName: "kube-api-access-x4tqs") pod "351d3a96-42eb-4959-93a6-38fd4eb2027b" (UID: "351d3a96-42eb-4959-93a6-38fd4eb2027b"). InnerVolumeSpecName "kube-api-access-x4tqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:38:28.475965 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.475943 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:38:28.480152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.480130 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4tqs\" (UniqueName: \"kubernetes.io/projected/351d3a96-42eb-4959-93a6-38fd4eb2027b-kube-api-access-x4tqs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:38:28.480152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.480152 2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/351d3a96-42eb-4959-93a6-38fd4eb2027b-tls-cert\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:38:28.580945 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.580916 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnf6f\" (UniqueName: \"kubernetes.io/projected/d1c4e633-3129-4e1f-a074-48604ae7edb6-kube-api-access-qnf6f\") pod \"d1c4e633-3129-4e1f-a074-48604ae7edb6\" (UID: \"d1c4e633-3129-4e1f-a074-48604ae7edb6\") " Jul 15 21:38:28.582901 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.582877 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c4e633-3129-4e1f-a074-48604ae7edb6-kube-api-access-qnf6f" (OuterVolumeSpecName: "kube-api-access-qnf6f") pod "d1c4e633-3129-4e1f-a074-48604ae7edb6" (UID: "d1c4e633-3129-4e1f-a074-48604ae7edb6"). InnerVolumeSpecName "kube-api-access-qnf6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:38:28.682092 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:28.682035 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnf6f\" (UniqueName: \"kubernetes.io/projected/d1c4e633-3129-4e1f-a074-48604ae7edb6-kube-api-access-qnf6f\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:38:29.214179 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.214129 2572 generic.go:356] "Generic (PLEG): container finished" podID="d1c4e633-3129-4e1f-a074-48604ae7edb6" containerID="03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b" exitCode=0 Jul 15 21:38:29.214179 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.214177 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-4k6xv" Jul 15 21:38:29.214373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.214194 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4k6xv" event={"ID":"d1c4e633-3129-4e1f-a074-48604ae7edb6","Type":"ContainerDied","Data":"03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b"} Jul 15 21:38:29.214373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.214243 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-4k6xv" event={"ID":"d1c4e633-3129-4e1f-a074-48604ae7edb6","Type":"ContainerDied","Data":"b8f7eec0de93380ad61ebef1f5501e6ad62cb32312dd8b7ebc5892d6753ca33a"} Jul 15 21:38:29.214373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.214258 2572 scope.go:117] "RemoveContainer" containerID="03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b" Jul 15 21:38:29.214373 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.214312 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-6jxpp" Jul 15 21:38:29.222717 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.222704 2572 scope.go:117] "RemoveContainer" containerID="03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b" Jul 15 21:38:29.222983 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:38:29.222963 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b\": container with ID starting with 03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b not found: ID does not exist" containerID="03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b" Jul 15 21:38:29.223044 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.222990 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b"} err="failed to get container status \"03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b\": rpc error: code = NotFound desc = could not find container \"03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b\": container with ID starting with 03955e82982e09817229d19f865c2a550824f2fdfbbd76b39303f3a128f7000b not found: ID does not exist" Jul 15 21:38:29.243954 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.243924 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-6jxpp"] Jul 15 21:38:29.246049 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.246009 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-6jxpp"] Jul 15 21:38:29.256321 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.256282 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4k6xv"] Jul 15 21:38:29.258139 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:29.258118 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-4k6xv"] Jul 15 21:38:30.061588 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:30.061557 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351d3a96-42eb-4959-93a6-38fd4eb2027b" path="/var/lib/kubelet/pods/351d3a96-42eb-4959-93a6-38fd4eb2027b/volumes" Jul 15 21:38:30.061926 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:30.061753 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c4e633-3129-4e1f-a074-48604ae7edb6" path="/var/lib/kubelet/pods/d1c4e633-3129-4e1f-a074-48604ae7edb6/volumes" Jul 15 21:38:32.925301 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:32.925270 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.024567 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:33.024535 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.089896 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:33.089861 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:33.187254 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:33.187187 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:36.375031 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:36.374999 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:36.484944 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:36.484913 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:41.791536 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:41.791497 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:41.891505 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:41.891476 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:46.190427 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:46.190391 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:46.634770 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:46.634737 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:48.048084 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:48.048046 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:48.110416 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:48.110385 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:38:59.851384 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:38:59.851307 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:08.061306 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:08.061273 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:08.158426 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:08.158399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:08.556609 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:08.556578 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:09.020126 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:09.020094 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:09.107231 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:09.107200 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:09.221391 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:09.221362 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:39:09.267623 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:09.267598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:19.808764 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:19.808734 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:21.212904 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:21.212868 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:22.907263 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:22.907232 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:38.730260 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:38.730224 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:39.337987 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:39.337951 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:39:41.694122 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:41.694087 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:42.625785 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:42.625745 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:39:49.908608 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:49.908574 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:51.129251 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:51.129218 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:39:54.014452 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:54.014416 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:58.179572 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.179541 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5"] Jul 15 21:39:58.179919 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.179831 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c4e633-3129-4e1f-a074-48604ae7edb6" containerName="authorino" Jul 15 21:39:58.179919 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.179851 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c4e633-3129-4e1f-a074-48604ae7edb6" containerName="authorino" Jul 15 21:39:58.179919 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.179900 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c4e633-3129-4e1f-a074-48604ae7edb6" containerName="authorino" Jul 15 21:39:58.182748 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.182727 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.185205 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.185180 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kpcw4\"" Jul 15 21:39:58.185431 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.185409 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 21:39:58.186347 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.186316 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 21:39:58.186454 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.186384 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 21:39:58.192845 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.192822 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5"] Jul 15 21:39:58.221027 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.221004 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:39:58.266045 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.266020 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77493385-61ce-456a-8cd5-9d5bd3239489-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.266151 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.266057 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.266151 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.266076 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtrt\" (UniqueName: \"kubernetes.io/projected/77493385-61ce-456a-8cd5-9d5bd3239489-kube-api-access-tvtrt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.266151 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.266096 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.266323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.266149 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.266323 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.266225 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367333 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367298 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367491 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367340 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367491 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77493385-61ce-456a-8cd5-9d5bd3239489-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367491 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367405 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367491 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367424 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtrt\" (UniqueName: \"kubernetes.io/projected/77493385-61ce-456a-8cd5-9d5bd3239489-kube-api-access-tvtrt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367491 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367444 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367777 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367754 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367831 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367783 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.367876 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.367848 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.369590 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.369572 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77493385-61ce-456a-8cd5-9d5bd3239489-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.369847 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.369831 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77493385-61ce-456a-8cd5-9d5bd3239489-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.374736 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.374716 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtrt\" (UniqueName: \"kubernetes.io/projected/77493385-61ce-456a-8cd5-9d5bd3239489-kube-api-access-tvtrt\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-srrb5\" (UID: \"77493385-61ce-456a-8cd5-9d5bd3239489\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.495116 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.495093 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:39:58.618357 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.618333 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5"] Jul 15 21:39:58.619775 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:39:58.619751 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77493385_61ce_456a_8cd5_9d5bd3239489.slice/crio-ccfafe6a51ff813c29e031353fa2cd8020d132224dd4bb81739bf8b7afd9d7a9 WatchSource:0}: Error finding container ccfafe6a51ff813c29e031353fa2cd8020d132224dd4bb81739bf8b7afd9d7a9: Status 404 returned error can't find the container with id ccfafe6a51ff813c29e031353fa2cd8020d132224dd4bb81739bf8b7afd9d7a9 Jul 15 21:39:58.633654 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:39:58.633628 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" event={"ID":"77493385-61ce-456a-8cd5-9d5bd3239489","Type":"ContainerStarted","Data":"ccfafe6a51ff813c29e031353fa2cd8020d132224dd4bb81739bf8b7afd9d7a9"} Jul 15 21:40:00.028415 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:00.028383 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:40:00.645432 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:00.645389 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" event={"ID":"77493385-61ce-456a-8cd5-9d5bd3239489","Type":"ContainerStarted","Data":"d0fb53e624d2a8f8806449fa5bf26748c297c37b88831f637e57ba076557ea57"} Jul 15 21:40:00.669571 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:00.669525 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" podStartSLOduration=1.6750145189999999 podStartE2EDuration="2.669510385s" totalImagesPullingTime="994.495866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:40:00.66828744 +0000 UTC m=+597.248884507" watchObservedRunningTime="2026-07-15 21:40:00.669510385 +0000 UTC m=+597.250107412" Jul 15 21:40:03.870911 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:03.870880 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:40:03.871443 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:03.871426 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:40:08.495517 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:08.495480 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:40:08.498080 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:08.495537 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:40:08.507601 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:08.507582 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:40:08.701070 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:08.701042 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-srrb5" Jul 15 21:40:11.315427 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:11.315398 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:14.042921 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:14.042882 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:40:56.340401 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.340325 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b68dcc544-lfb7k"] Jul 15 21:40:56.344944 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.344925 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.352828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.352805 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b68dcc544-lfb7k"] Jul 15 21:40:56.429009 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.428974 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fb7d29a-977c-4c7b-b48a-1d99d3c99c58-tls-cert\") pod \"authorino-5b68dcc544-lfb7k\" (UID: \"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58\") " pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.429203 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.429028 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ndl\" (UniqueName: \"kubernetes.io/projected/6fb7d29a-977c-4c7b-b48a-1d99d3c99c58-kube-api-access-m5ndl\") pod \"authorino-5b68dcc544-lfb7k\" (UID: \"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58\") " pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.529872 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.529832 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fb7d29a-977c-4c7b-b48a-1d99d3c99c58-tls-cert\") pod \"authorino-5b68dcc544-lfb7k\" (UID: \"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58\") " pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.530075 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.529894 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ndl\" (UniqueName: \"kubernetes.io/projected/6fb7d29a-977c-4c7b-b48a-1d99d3c99c58-kube-api-access-m5ndl\") pod \"authorino-5b68dcc544-lfb7k\" (UID: \"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58\") " pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.532312 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.532290 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6fb7d29a-977c-4c7b-b48a-1d99d3c99c58-tls-cert\") pod \"authorino-5b68dcc544-lfb7k\" (UID: \"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58\") " pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.536845 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.536823 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ndl\" (UniqueName: \"kubernetes.io/projected/6fb7d29a-977c-4c7b-b48a-1d99d3c99c58-kube-api-access-m5ndl\") pod \"authorino-5b68dcc544-lfb7k\" (UID: \"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58\") " pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.654566 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.654475 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b68dcc544-lfb7k" Jul 15 21:40:56.780371 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.780326 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b68dcc544-lfb7k"] Jul 15 21:40:56.782788 ip-10-0-135-178 kubenswrapper[2572]: W0715 21:40:56.782763 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fb7d29a_977c_4c7b_b48a_1d99d3c99c58.slice/crio-fad5f6381456110a492bab8e74ff89a8127fcfee58dc9c9e396c531d192c4cbf WatchSource:0}: Error finding container fad5f6381456110a492bab8e74ff89a8127fcfee58dc9c9e396c531d192c4cbf: Status 404 returned error can't find the container with id fad5f6381456110a492bab8e74ff89a8127fcfee58dc9c9e396c531d192c4cbf Jul 15 21:40:56.784443 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.784427 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 21:40:56.919548 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:56.919464 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b68dcc544-lfb7k" event={"ID":"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58","Type":"ContainerStarted","Data":"fad5f6381456110a492bab8e74ff89a8127fcfee58dc9c9e396c531d192c4cbf"} Jul 15 21:40:57.927625 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:57.927588 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b68dcc544-lfb7k" event={"ID":"6fb7d29a-977c-4c7b-b48a-1d99d3c99c58","Type":"ContainerStarted","Data":"0a28268535bd08055dcca79c37301a29f5042549a62349ca8eed526151e02d48"} Jul 15 21:40:57.948252 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:57.948196 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5b68dcc544-lfb7k" podStartSLOduration=1.575763948 podStartE2EDuration="1.948159231s" totalImagesPullingTime="372.395283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 21:40:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 21:40:57.945907856 +0000 UTC m=+654.526504879" watchObservedRunningTime="2026-07-15 21:40:57.948159231 +0000 UTC m=+654.528756259" Jul 15 21:40:57.978213 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:57.973350 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f6c9c766f-47tsj"] Jul 15 21:40:57.978213 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:57.973551 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" podUID="2116e0eb-c2a8-40d8-83b8-79b5873000d6" containerName="authorino" containerID="cri-o://4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d" gracePeriod=30 Jul 15 21:40:58.228622 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.228588 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:40:58.244626 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.244586 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116e0eb-c2a8-40d8-83b8-79b5873000d6-tls-cert\") pod \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " Jul 15 21:40:58.244782 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.244640 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdfcm\" (UniqueName: \"kubernetes.io/projected/2116e0eb-c2a8-40d8-83b8-79b5873000d6-kube-api-access-hdfcm\") pod \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\" (UID: \"2116e0eb-c2a8-40d8-83b8-79b5873000d6\") " Jul 15 21:40:58.250217 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.248280 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2116e0eb-c2a8-40d8-83b8-79b5873000d6-kube-api-access-hdfcm" (OuterVolumeSpecName: "kube-api-access-hdfcm") pod "2116e0eb-c2a8-40d8-83b8-79b5873000d6" (UID: "2116e0eb-c2a8-40d8-83b8-79b5873000d6"). InnerVolumeSpecName "kube-api-access-hdfcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 21:40:58.256335 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.256306 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2116e0eb-c2a8-40d8-83b8-79b5873000d6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "2116e0eb-c2a8-40d8-83b8-79b5873000d6" (UID: "2116e0eb-c2a8-40d8-83b8-79b5873000d6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 21:40:58.345952 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.345909 2572 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2116e0eb-c2a8-40d8-83b8-79b5873000d6-tls-cert\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:40:58.345952 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.345951 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfcm\" (UniqueName: \"kubernetes.io/projected/2116e0eb-c2a8-40d8-83b8-79b5873000d6-kube-api-access-hdfcm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 21:40:58.493054 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.492955 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:58.581528 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.581487 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:58.934437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.934353 2572 generic.go:356] "Generic (PLEG): container finished" podID="2116e0eb-c2a8-40d8-83b8-79b5873000d6" containerID="4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d" exitCode=0 Jul 15 21:40:58.934437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.934397 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" Jul 15 21:40:58.934888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.934443 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" event={"ID":"2116e0eb-c2a8-40d8-83b8-79b5873000d6","Type":"ContainerDied","Data":"4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d"} Jul 15 21:40:58.934888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.934490 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f6c9c766f-47tsj" event={"ID":"2116e0eb-c2a8-40d8-83b8-79b5873000d6","Type":"ContainerDied","Data":"6ef7b38cbb80d2cf4644cf859249e6c7c625e00daf9af9f121f45e4a5b5ed618"} Jul 15 21:40:58.934888 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.934509 2572 scope.go:117] "RemoveContainer" containerID="4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d" Jul 15 21:40:58.945331 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.945313 2572 scope.go:117] "RemoveContainer" containerID="4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d" Jul 15 21:40:58.945793 ip-10-0-135-178 kubenswrapper[2572]: E0715 21:40:58.945726 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d\": container with ID starting with 4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d not found: ID does not exist" containerID="4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d" Jul 15 21:40:58.945793 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.945762 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d"} err="failed to get container status \"4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d\": rpc error: code = NotFound desc = could not find container \"4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d\": container with ID starting with 4db416ca48ceebe931bfdf3febd35d548b424fc4b87a65e6da43df81a5495c0d not found: ID does not exist" Jul 15 21:40:58.983956 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:58.983914 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f6c9c766f-47tsj"] Jul 15 21:40:59.004254 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:59.004219 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f6c9c766f-47tsj"] Jul 15 21:40:59.669691 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:59.669653 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:40:59.773388 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:40:59.773353 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:41:00.062725 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:41:00.062696 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2116e0eb-c2a8-40d8-83b8-79b5873000d6" path="/var/lib/kubelet/pods/2116e0eb-c2a8-40d8-83b8-79b5873000d6/volumes" Jul 15 21:41:27.631052 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:41:27.631020 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:41:37.438142 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:41:37.438105 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:41:48.530611 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:41:48.530577 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:41:58.232891 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:41:58.232806 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:42:08.426181 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:08.426134 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:42:10.488556 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:10.488519 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:12.416670 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:12.416636 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:35.630242 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:35.630210 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:42:49.530977 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:49.530941 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:42:54.168953 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:54.168923 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:54.267871 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:54.267842 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:42:54.446190 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:42:54.446139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:00.212553 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:00.212516 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:00.269000 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:00.268963 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:00.359333 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:00.359303 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:00.465469 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:00.465395 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:00.574343 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:00.574310 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:06.149024 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:06.148993 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:06.249547 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:06.249512 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:23.427055 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:23.427020 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:43:32.723706 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:32.723628 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:43:41.229047 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:41.229015 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:43:43.880855 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:43.880826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:43:51.032349 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:43:51.032313 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:44:01.325872 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:44:01.325830 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:44:10.629737 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:44:10.629703 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:44:21.629306 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:44:21.629270 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:44:31.329797 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:44:31.329761 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:44:53.929268 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:44:53.929233 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:44:55.414283 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:44:55.414247 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:03.900515 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:03.900486 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:45:03.900964 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:03.900946 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:45:06.227802 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:06.227768 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:45:12.207004 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:12.206972 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:45:29.244648 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:29.244612 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:45:40.029600 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:40.029560 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:45:54.938013 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:45:54.936179 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:46:05.939898 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:46:05.939862 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:46:15.629790 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:46:15.629744 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:46:25.027043 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:46:25.027005 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:46:34.626117 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:46:34.626034 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:47:54.508238 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:47:54.508187 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:47:54.554819 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:47:54.554790 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:48:22.035614 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:48:22.035578 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:48:33.130889 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:48:33.130854 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:48:41.727160 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:48:41.727125 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:48:54.136284 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:48:54.136245 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:49:05.043561 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:49:05.043520 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:49:15.828195 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:49:15.826926 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:49:24.529117 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:49:24.529080 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:49:35.935906 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:49:35.935827 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:49:46.929708 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:49:46.929669 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:49:55.722849 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:49:55.722818 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:50:03.931204 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:03.931155 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:50:03.933403 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:03.933378 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:50:06.429702 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:06.429669 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:50:07.692140 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:07.692102 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:50:28.730708 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:28.730675 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:50:37.734523 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:37.734480 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:50:49.232756 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:50:49.232717 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:51:01.728012 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:51:01.727931 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:51:18.433152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:51:18.433118 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:51:29.038793 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:51:29.038758 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:51:38.231104 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:51:38.231067 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:51:48.727270 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:51:48.727235 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:51:57.837584 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:51:57.837544 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:52:07.328887 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:52:07.328853 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:52:20.130206 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:52:20.130150 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:52:29.331828 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:52:29.331731 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:52:38.930898 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:52:38.930863 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:52:50.430228 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:52:50.430197 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:53:00.836217 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:00.836182 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:53:07.727872 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:07.727834 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:53:24.437942 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:24.437903 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:53:37.238384 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:37.238347 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:53:46.531131 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:46.531093 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:53:49.986360 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:49.986326 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:53:52.862055 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:52.861978 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:53:56.825993 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:56.825912 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:53:59.933435 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:53:59.933402 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:54:05.082285 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:05.082238 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:05.650396 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:05.650364 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:09.225357 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:09.225318 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:54:10.232853 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:10.232821 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:13.299108 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:13.299073 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:13.383787 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:13.383742 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:13.742817 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:13.742776 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:18.723503 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:18.723469 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:54:19.365719 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:19.365682 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:19.468430 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:19.468391 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:19.594136 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:19.594097 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:19.773105 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:19.773069 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:20.686374 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:20.686328 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:20.789055 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:20.789020 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:20.883661 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:20.883625 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:21.271995 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:21.271959 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:21.372104 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:21.372073 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:21.476375 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:21.476344 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:23.173778 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:23.173728 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:54:25.837468 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:54:25.837433 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:55:03.960806 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:03.960778 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:55:03.965307 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:03.965284 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 21:55:26.470515 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:26.470439 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:28.330049 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:28.330014 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:55:31.061134 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:31.061098 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:36.792428 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:36.792392 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:36.889776 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:36.889744 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:41.162942 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:41.162910 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:41.624925 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:41.624886 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:41.957807 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:41.957775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:42.060378 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:42.060346 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:43.000947 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:43.000911 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:43.117869 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:43.117826 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:43.612381 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:43.612348 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:43.710213 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:43.710155 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:45.425171 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:45.425126 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:55:47.545099 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:47.545071 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:47.645433 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:47.645399 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:55:50.619044 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:55:50.619016 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:47.698619 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:47.698587 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:47.813880 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:47.813847 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:48.065548 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:48.065461 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:50.436518 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:50.436487 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:56:53.651313 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:53.651271 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:53.715673 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:53.715637 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:53.874214 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:53.874182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:54.371349 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:54.371311 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:54.482962 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:54.482925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:54.984049 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:54.984016 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:55.083497 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:55.083468 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:55.182977 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:55.182939 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:55.443986 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:55.443948 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:55.644954 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:55.644921 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:55.745664 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:55.745579 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:57.215589 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:57.215560 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:57.688513 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:57.688475 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:56:58.029538 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:56:58.029502 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:57:33.604549 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:57:33.604518 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:57:33.705324 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:57:33.705289 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:00.137461 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:00.137422 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:00.237979 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:00.237946 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:09.930288 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:09.930251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:12.630594 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:12.630554 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:58:14.564489 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:14.564446 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:20.107446 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:20.107380 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:20.186653 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:20.186613 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:24.760002 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:24.759956 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:25.374552 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:25.374517 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:25.473987 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:25.473950 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:25.574789 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:25.574753 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:27.027152 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:27.027064 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:32.667461 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:32.667423 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:32.769611 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:32.769572 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:32.867726 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:32.867694 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:32.971153 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:32.971098 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:33.407928 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:33.407840 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:36.766762 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:36.766725 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:58:59.433437 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:58:59.433402 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:05.163213 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:05.163157 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:05.226669 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:05.226633 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:05.299840 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:05.299784 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:05.398593 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:05.398555 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:21.410224 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:21.410160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:26.898399 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:26.898363 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:27.082669 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:27.082628 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:27.191645 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:27.191617 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:27.581200 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:27.581084 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:37.631648 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:37.631604 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:38.454957 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:38.454923 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:44.091381 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:44.091344 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:44.204251 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:44.204214 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:44.292021 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:44.291984 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:44.391691 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:44.391597 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:44.820807 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:44.820771 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:44.914921 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:44.914891 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:45.177362 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:45.177283 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:45.277021 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:45.276985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:45.379347 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:45.379309 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:47.019997 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:47.019960 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:47.725745 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:47.725703 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:59:52.734195 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:52.734135 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:52.834251 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:52.834219 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:53.418714 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:53.418679 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:57.028397 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:57.028316 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 21:59:59.028740 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:59.028701 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:59.129028 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:59.128993 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:59.266900 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:59.266862 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:59.595327 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:59.595290 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 21:59:59.982573 ip-10-0-135-178 kubenswrapper[2572]: I0715 21:59:59.982538 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:02.557660 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:02.557621 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:02.684129 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:02.684090 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:02.938237 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:02.938119 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:03.042623 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:03.042586 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:03.525075 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:03.525036 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:03.801528 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:03.801441 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:03.989677 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:03.989652 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 22:00:04.001513 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:04.001487 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 22:00:07.011408 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:07.011367 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:07.674259 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:07.674220 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:07.773439 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:07.773405 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:08.036636 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:08.036594 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:00:08.127507 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:08.127473 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:13.986504 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:13.986455 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:14.086039 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:14.085996 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:24.719821 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:24.719775 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:27.730971 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:27.730934 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:00:29.370494 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:29.370462 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:34.987950 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:34.987898 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:35.044788 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:35.044753 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:40.043231 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:40.043194 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:40.291414 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:40.291384 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:40.413143 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:40.413059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:40.510687 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:40.510648 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:42.000246 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:42.000210 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:46.265293 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:46.265260 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:46.456181 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:46.456139 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:46.526568 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:46.526482 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:46.627609 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:46.627571 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:52.153095 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:52.153053 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:52.252655 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:52.252621 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:59.847069 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:59.847025 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:00:59.949934 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:00:59.949891 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:00.038497 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:00.038463 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:00.780354 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:00.780316 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:00.879844 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:00.879792 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:00.982776 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:00.982743 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:02.366105 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:02.366068 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:03.840433 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:03.840397 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:01:07.883062 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:07.883009 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:07.939205 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:07.939178 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:08.572098 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:08.572059 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:14.208933 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:14.208896 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:14.321340 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:14.321302 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:14.408709 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:14.408674 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:15.638500 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:15.638463 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:01:24.957042 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:24.957004 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:25.332659 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:25.332571 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:47.851366 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:47.851319 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:51.912346 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:51.912308 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:52.012076 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:52.012044 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:52.257947 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:52.257909 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:52.570509 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:52.570419 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:52.873597 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:52.873506 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:55.819553 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:55.819508 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:55.918741 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:55.918707 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:56.070568 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:56.070459 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:56.534729 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:56.534687 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:56.637806 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:56.637764 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:01:56.738910 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:01:56.738870 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:00.362603 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:00.362570 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:00.712306 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:00.712268 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:02.527477 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:02.527436 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:02.574572 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:02.574527 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:03.828898 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:03.828863 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:02:04.288815 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:04.288781 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:04.388027 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:04.387985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:04.571656 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:04.571555 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:04.778459 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:04.778430 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:05.160822 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:05.160785 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:05.262823 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:05.262780 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:08.802035 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:08.801998 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:17.367603 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:17.367566 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:17.468145 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:17.468109 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:18.223906 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:18.223873 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:02:19.885584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.885547 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg"] Jul 15 22:02:19.885962 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.885853 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2116e0eb-c2a8-40d8-83b8-79b5873000d6" containerName="authorino" Jul 15 22:02:19.885962 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.885864 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2116e0eb-c2a8-40d8-83b8-79b5873000d6" containerName="authorino" Jul 15 22:02:19.885962 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.885930 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2116e0eb-c2a8-40d8-83b8-79b5873000d6" containerName="authorino" Jul 15 22:02:19.888743 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.888725 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:19.892021 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.891995 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4081d1\"/\"openshift-service-ca.crt\"" Jul 15 22:02:19.893101 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.893085 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4081d1\"/\"test-model-4081d1-kserve-self-signed-certs\"" Jul 15 22:02:19.893242 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.893105 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-4081d1\"/\"kube-root-ca.crt\"" Jul 15 22:02:19.893242 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.893105 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-4081d1\"/\"default-dockercfg-rh5qz\"" Jul 15 22:02:19.898063 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.898040 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg"] Jul 15 22:02:19.903282 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.903261 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:19.967076 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.967037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-model-cache\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:19.967269 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.967093 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-tmp-dir\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:19.967269 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.967146 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-home\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:19.967269 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.967192 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-dshm\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:19.967269 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.967217 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537b1fdd-c268-46fe-917a-4329bf619e8e-tls-certs\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:19.967269 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:19.967243 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhql7\" (UniqueName: \"kubernetes.io/projected/537b1fdd-c268-46fe-917a-4329bf619e8e-kube-api-access-nhql7\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068022 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.067986 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-home\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068210 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068035 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-dshm\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068210 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068064 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537b1fdd-c268-46fe-917a-4329bf619e8e-tls-certs\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068210 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068095 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhql7\" (UniqueName: \"kubernetes.io/projected/537b1fdd-c268-46fe-917a-4329bf619e8e-kube-api-access-nhql7\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068210 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068157 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-model-cache\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068433 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068229 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-tmp-dir\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068495 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068476 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-home\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068580 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-tmp-dir\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.068637 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.068592 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-model-cache\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.070339 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.070319 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-dshm\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.070630 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.070611 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537b1fdd-c268-46fe-917a-4329bf619e8e-tls-certs\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.076375 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.076357 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhql7\" (UniqueName: \"kubernetes.io/projected/537b1fdd-c268-46fe-917a-4329bf619e8e-kube-api-access-nhql7\") pod \"test-model-4081d1-kserve-659b598597-pmqgg\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.198220 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.198189 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:20.329744 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:20.329718 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg"] Jul 15 22:02:20.332115 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:02:20.332090 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod537b1fdd_c268_46fe_917a_4329bf619e8e.slice/crio-89e64894d497cb50ceda768fa7113dc3ba00fd704df894d39996a32dfb365797 WatchSource:0}: Error finding container 89e64894d497cb50ceda768fa7113dc3ba00fd704df894d39996a32dfb365797: Status 404 returned error can't find the container with id 89e64894d497cb50ceda768fa7113dc3ba00fd704df894d39996a32dfb365797 Jul 15 22:02:21.075755 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:21.075715 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" event={"ID":"537b1fdd-c268-46fe-917a-4329bf619e8e","Type":"ContainerStarted","Data":"80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf"} Jul 15 22:02:21.076212 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:21.075763 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" event={"ID":"537b1fdd-c268-46fe-917a-4329bf619e8e","Type":"ContainerStarted","Data":"89e64894d497cb50ceda768fa7113dc3ba00fd704df894d39996a32dfb365797"} Jul 15 22:02:21.091344 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:21.091291 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" podStartSLOduration=2.091274145 podStartE2EDuration="2.091274145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:02:21.09011196 +0000 UTC m=+1937.670709003" watchObservedRunningTime="2026-07-15 22:02:21.091274145 +0000 UTC m=+1937.671871173" Jul 15 22:02:27.076749 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:27.076709 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:27.176877 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:27.176822 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:29.740749 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:29.740713 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:02:30.198601 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:30.198561 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:30.198816 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:30.198623 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:30.211030 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:30.211000 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:31.133284 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:31.133253 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:02:34.479664 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:34.479600 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:34.579977 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:34.579935 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:43.488236 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.488199 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b"] Jul 15 22:02:43.491416 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.491396 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.494074 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.494043 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a6de7e\"/\"openshift-service-ca.crt\"" Jul 15 22:02:43.495049 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.495027 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a6de7e\"/\"default-dockercfg-ffksd\"" Jul 15 22:02:43.495178 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.495062 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a6de7e\"/\"kube-root-ca.crt\"" Jul 15 22:02:43.495364 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.495347 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a6de7e\"/\"test-model-a6de7e-kserve-self-signed-certs\"" Jul 15 22:02:43.501233 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.501212 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b"] Jul 15 22:02:43.517354 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.517329 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:02:43.666018 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.665982 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-home\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.666207 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.666034 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxx4c\" (UniqueName: \"kubernetes.io/projected/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-kube-api-access-qxx4c\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.666207 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.666117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tls-certs\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.666207 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.666151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-model-cache\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.666207 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.666190 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-dshm\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.666207 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.666208 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tmp-dir\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766647 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766565 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-home\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766647 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766612 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxx4c\" (UniqueName: \"kubernetes.io/projected/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-kube-api-access-qxx4c\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766834 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766789 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tls-certs\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766834 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-model-cache\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766939 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766845 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-dshm\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766997 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766951 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tmp-dir\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.766997 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.766956 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-home\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.767234 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.767214 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-model-cache\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.767315 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.767242 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tmp-dir\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.769055 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.769030 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-dshm\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.769262 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.769243 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tls-certs\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.774158 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.774136 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxx4c\" (UniqueName: \"kubernetes.io/projected/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-kube-api-access-qxx4c\") pod \"test-model-a6de7e-kserve-7585d7487c-b6p9b\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.802434 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.802413 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:43.936239 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:43.936213 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b"] Jul 15 22:02:43.937822 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:02:43.937794 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f3fcef5_37fc_4b45_b4ed_f6df06fa376f.slice/crio-704ec5e8a4a95851a99f2b9e8ba50ed7e2fcafa4d94a326368c05e8147ee0db8 WatchSource:0}: Error finding container 704ec5e8a4a95851a99f2b9e8ba50ed7e2fcafa4d94a326368c05e8147ee0db8: Status 404 returned error can't find the container with id 704ec5e8a4a95851a99f2b9e8ba50ed7e2fcafa4d94a326368c05e8147ee0db8 Jul 15 22:02:44.184462 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:44.184362 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" event={"ID":"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f","Type":"ContainerStarted","Data":"30216b5971c727a23f558f8dd6c5443e89aec35c75b728b4d698cf3dd1023516"} Jul 15 22:02:44.184462 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:44.184405 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" event={"ID":"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f","Type":"ContainerStarted","Data":"704ec5e8a4a95851a99f2b9e8ba50ed7e2fcafa4d94a326368c05e8147ee0db8"} Jul 15 22:02:44.202109 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:44.202064 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" podStartSLOduration=1.202047889 podStartE2EDuration="1.202047889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:02:44.201268439 +0000 UTC m=+1960.781865469" watchObservedRunningTime="2026-07-15 22:02:44.202047889 +0000 UTC m=+1960.782644918" Jul 15 22:02:47.931637 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:47.931601 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:02:53.803291 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:53.803250 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:53.803817 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:53.803313 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:53.815728 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:53.815700 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:02:54.242827 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:02:54.242800 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:03:02.128303 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:02.128222 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:03:02.268778 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:02.268742 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg"] Jul 15 22:03:02.287723 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:02.287693 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:03.158753 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:03.158720 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b"] Jul 15 22:03:03.179431 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:03.179398 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:03.938259 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:03.938223 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:04.039306 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:04.039276 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:13.789334 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:13.789295 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:14.533113 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:14.533082 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:03:17.290922 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.290866 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" podUID="537b1fdd-c268-46fe-917a-4329bf619e8e" containerName="main" containerID="cri-o://80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf" gracePeriod=45 Jul 15 22:03:17.535391 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.535364 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:03:17.626060 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.625952 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-tmp-dir\") pod \"537b1fdd-c268-46fe-917a-4329bf619e8e\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " Jul 15 22:03:17.626060 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.625999 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-home\") pod \"537b1fdd-c268-46fe-917a-4329bf619e8e\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " Jul 15 22:03:17.626060 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626033 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537b1fdd-c268-46fe-917a-4329bf619e8e-tls-certs\") pod \"537b1fdd-c268-46fe-917a-4329bf619e8e\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " Jul 15 22:03:17.626379 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626081 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-dshm\") pod \"537b1fdd-c268-46fe-917a-4329bf619e8e\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " Jul 15 22:03:17.626379 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626104 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhql7\" (UniqueName: \"kubernetes.io/projected/537b1fdd-c268-46fe-917a-4329bf619e8e-kube-api-access-nhql7\") pod \"537b1fdd-c268-46fe-917a-4329bf619e8e\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " Jul 15 22:03:17.626379 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626155 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-model-cache\") pod \"537b1fdd-c268-46fe-917a-4329bf619e8e\" (UID: \"537b1fdd-c268-46fe-917a-4329bf619e8e\") " Jul 15 22:03:17.626379 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626335 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "537b1fdd-c268-46fe-917a-4329bf619e8e" (UID: "537b1fdd-c268-46fe-917a-4329bf619e8e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:17.626540 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626343 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-home" (OuterVolumeSpecName: "home") pod "537b1fdd-c268-46fe-917a-4329bf619e8e" (UID: "537b1fdd-c268-46fe-917a-4329bf619e8e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:17.626540 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626436 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "537b1fdd-c268-46fe-917a-4329bf619e8e" (UID: "537b1fdd-c268-46fe-917a-4329bf619e8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:17.626540 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.626459 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:17.628464 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.628441 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-dshm" (OuterVolumeSpecName: "dshm") pod "537b1fdd-c268-46fe-917a-4329bf619e8e" (UID: "537b1fdd-c268-46fe-917a-4329bf619e8e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:17.628539 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.628468 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/537b1fdd-c268-46fe-917a-4329bf619e8e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "537b1fdd-c268-46fe-917a-4329bf619e8e" (UID: "537b1fdd-c268-46fe-917a-4329bf619e8e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:03:17.628579 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.628557 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/537b1fdd-c268-46fe-917a-4329bf619e8e-kube-api-access-nhql7" (OuterVolumeSpecName: "kube-api-access-nhql7") pod "537b1fdd-c268-46fe-917a-4329bf619e8e" (UID: "537b1fdd-c268-46fe-917a-4329bf619e8e"). InnerVolumeSpecName "kube-api-access-nhql7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:03:17.727845 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.727810 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:17.727845 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.727838 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/537b1fdd-c268-46fe-917a-4329bf619e8e-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:17.727845 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.727848 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:17.727845 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.727857 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhql7\" (UniqueName: \"kubernetes.io/projected/537b1fdd-c268-46fe-917a-4329bf619e8e-kube-api-access-nhql7\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:17.728111 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:17.727866 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/537b1fdd-c268-46fe-917a-4329bf619e8e-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:18.179502 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.179465 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" podUID="3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" containerName="main" containerID="cri-o://30216b5971c727a23f558f8dd6c5443e89aec35c75b728b4d698cf3dd1023516" gracePeriod=45 Jul 15 22:03:18.355551 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.355522 2572 generic.go:356] "Generic (PLEG): container finished" podID="3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" containerID="30216b5971c727a23f558f8dd6c5443e89aec35c75b728b4d698cf3dd1023516" exitCode=0 Jul 15 22:03:18.356010 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.355598 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" event={"ID":"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f","Type":"ContainerDied","Data":"30216b5971c727a23f558f8dd6c5443e89aec35c75b728b4d698cf3dd1023516"} Jul 15 22:03:18.357019 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.356985 2572 generic.go:356] "Generic (PLEG): container finished" podID="537b1fdd-c268-46fe-917a-4329bf619e8e" containerID="80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf" exitCode=0 Jul 15 22:03:18.357134 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.357035 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" Jul 15 22:03:18.357134 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.357043 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" event={"ID":"537b1fdd-c268-46fe-917a-4329bf619e8e","Type":"ContainerDied","Data":"80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf"} Jul 15 22:03:18.357134 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.357081 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg" event={"ID":"537b1fdd-c268-46fe-917a-4329bf619e8e","Type":"ContainerDied","Data":"89e64894d497cb50ceda768fa7113dc3ba00fd704df894d39996a32dfb365797"} Jul 15 22:03:18.357134 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.357102 2572 scope.go:117] "RemoveContainer" containerID="80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf" Jul 15 22:03:18.367981 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.367959 2572 scope.go:117] "RemoveContainer" containerID="80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf" Jul 15 22:03:18.368278 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:03:18.368254 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf\": container with ID starting with 80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf not found: ID does not exist" containerID="80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf" Jul 15 22:03:18.368352 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.368290 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf"} err="failed to get container status \"80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf\": rpc error: code = NotFound desc = could not find container \"80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf\": container with ID starting with 80ec599bef29053c704cf7565287a955c267007d5df60f7e5f7de7970c6132bf not found: ID does not exist" Jul 15 22:03:18.380465 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.376104 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg"] Jul 15 22:03:18.380465 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.379450 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-4081d1/test-model-4081d1-kserve-659b598597-pmqgg"] Jul 15 22:03:18.417564 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.417354 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:18.433215 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.433187 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:03:18.533532 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533500 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-dshm\") pod \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " Jul 15 22:03:18.533728 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533557 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-model-cache\") pod \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " Jul 15 22:03:18.533728 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533581 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tmp-dir\") pod \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " Jul 15 22:03:18.533728 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533603 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qxx4c\" (UniqueName: \"kubernetes.io/projected/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-kube-api-access-qxx4c\") pod \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " Jul 15 22:03:18.533728 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533627 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-home\") pod \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " Jul 15 22:03:18.533728 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533653 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tls-certs\") pod \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\" (UID: \"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f\") " Jul 15 22:03:18.533982 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533880 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" (UID: "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:18.533982 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533894 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" (UID: "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:18.533982 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.533912 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-home" (OuterVolumeSpecName: "home") pod "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" (UID: "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:18.535948 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.535924 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-kube-api-access-qxx4c" (OuterVolumeSpecName: "kube-api-access-qxx4c") pod "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" (UID: "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f"). InnerVolumeSpecName "kube-api-access-qxx4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:03:18.535948 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.535935 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" (UID: "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:03:18.536114 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.535946 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-dshm" (OuterVolumeSpecName: "dshm") pod "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" (UID: "3f3fcef5-37fc-4b45-b4ed-f6df06fa376f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:03:18.634584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.634503 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:18.634584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.634531 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:18.634584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.634563 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:18.634584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.634572 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxx4c\" (UniqueName: \"kubernetes.io/projected/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-kube-api-access-qxx4c\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:18.634584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.634582 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:18.634833 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:18.634592 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:03:19.365040 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:19.365003 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" event={"ID":"3f3fcef5-37fc-4b45-b4ed-f6df06fa376f","Type":"ContainerDied","Data":"704ec5e8a4a95851a99f2b9e8ba50ed7e2fcafa4d94a326368c05e8147ee0db8"} Jul 15 22:03:19.365040 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:19.365025 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b" Jul 15 22:03:19.365556 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:19.365052 2572 scope.go:117] "RemoveContainer" containerID="30216b5971c727a23f558f8dd6c5443e89aec35c75b728b4d698cf3dd1023516" Jul 15 22:03:19.385338 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:19.385305 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b"] Jul 15 22:03:19.388646 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:19.388624 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a6de7e/test-model-a6de7e-kserve-7585d7487c-b6p9b"] Jul 15 22:03:20.063344 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:20.063308 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" path="/var/lib/kubelet/pods/3f3fcef5-37fc-4b45-b4ed-f6df06fa376f/volumes" Jul 15 22:03:20.063715 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:20.063703 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="537b1fdd-c268-46fe-917a-4329bf619e8e" path="/var/lib/kubelet/pods/537b1fdd-c268-46fe-917a-4329bf619e8e/volumes" Jul 15 22:03:24.105583 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:24.105543 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:24.161117 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:24.161086 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:24.630834 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:24.630798 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:03:29.390614 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:29.390574 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:29.491991 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:29.491957 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:30.030147 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:30.030118 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:30.132316 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:30.132287 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:40.121482 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:40.121447 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:43.331644 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:43.331610 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:03:44.607515 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:44.607484 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:50.308963 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:50.308925 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:50.409558 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:50.409520 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:55.320025 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:55.319969 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:55.552790 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:55.552749 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:55.922747 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:55.922712 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:56.231153 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:56.231119 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:56.638556 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:56.638465 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:59.516690 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:59.516649 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:59.593700 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:59.593667 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:59.656470 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:59.656437 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:03:59.855488 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:03:59.855389 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:00.261631 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:00.261598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:00.365991 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:00.365954 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:04.131209 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:04.131160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:04.509342 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:04.509299 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:04.740964 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:04.740930 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:04:04.820060 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:04.819985 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:05.015733 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:05.015684 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:07.896593 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:07.896553 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:08.005454 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:08.005417 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:08.096286 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:08.096255 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:08.198711 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:08.198669 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:08.618188 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:08.618095 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:08.683602 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:08.683567 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:12.418423 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:12.418388 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:21.171921 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:21.171881 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:21.272291 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:21.272251 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:21.333836 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:21.333807 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:04:29.435746 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:29.435666 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:29.536576 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:29.536548 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:31.265424 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:31.265390 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:31.351872 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:31.351837 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:31.520540 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:31.520456 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:35.611341 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:35.611300 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:36.934518 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:36.934480 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:04:41.240935 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:41.240891 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:41.342369 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:41.342339 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:46.294218 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:46.294160 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:46.524013 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:46.523980 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:47.183660 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:47.183626 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:47.458629 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:47.458597 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:51.157313 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:51.157273 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:04:57.089673 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:04:57.089637 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:00.229046 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:00.229005 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:05:01.757042 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:01.757001 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:04.025787 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:04.025756 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 22:05:04.034430 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:04.034406 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 22:05:07.353559 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:07.353500 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:07.414720 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:07.414693 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:12.541674 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:12.541638 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:12.786916 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:12.786877 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:13.162516 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:13.162484 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:13.484475 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:13.484441 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:13.883648 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:13.883575 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:15.984395 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:15.984359 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:16.097027 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:16.096993 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:16.275621 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:16.275534 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:16.429533 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:16.429500 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:16.869772 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:16.869731 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:16.970266 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:16.970234 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:20.519004 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:20.518970 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:20.906325 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:20.906229 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:21.165567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:21.165478 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:21.438828 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:21.438736 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:22.432280 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:22.432246 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:05:22.610372 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:22.610340 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:24.313627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:24.313582 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:24.411865 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:24.411814 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:24.609022 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:24.608941 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:24.728592 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:24.728552 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:25.228494 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:25.228461 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:25.326046 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:25.326003 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:28.835904 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:28.835867 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:36.426813 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:36.426774 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:05:37.601612 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:37.601561 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:39.162916 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:39.162880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:41.693091 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693053 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt"] Jul 15 22:05:41.693567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693384 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" containerName="main" Jul 15 22:05:41.693567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693395 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" containerName="main" Jul 15 22:05:41.693567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693406 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="537b1fdd-c268-46fe-917a-4329bf619e8e" containerName="main" Jul 15 22:05:41.693567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693411 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="537b1fdd-c268-46fe-917a-4329bf619e8e" containerName="main" Jul 15 22:05:41.693567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693473 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3fcef5-37fc-4b45-b4ed-f6df06fa376f" containerName="main" Jul 15 22:05:41.693567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.693481 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="537b1fdd-c268-46fe-917a-4329bf619e8e" containerName="main" Jul 15 22:05:41.695483 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.695465 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.697980 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.697959 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8172c4\"/\"kube-root-ca.crt\"" Jul 15 22:05:41.699112 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.699077 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8172c4\"/\"auth-test-model-8172c4-kserve-self-signed-certs\"" Jul 15 22:05:41.699237 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.699141 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8172c4\"/\"default-dockercfg-z4kvr\"" Jul 15 22:05:41.699237 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.699158 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-8172c4\"/\"openshift-service-ca.crt\"" Jul 15 22:05:41.705136 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.705114 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt"] Jul 15 22:05:41.727694 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.727666 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:41.747793 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.747758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-model-cache\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.747901 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.747819 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph2bb\" (UniqueName: \"kubernetes.io/projected/0f911511-7cb5-4eff-a4c8-628139d6ca79-kube-api-access-ph2bb\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.747901 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.747882 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-home\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.748006 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.747906 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-tmp-dir\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.748006 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.747930 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-dshm\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.748006 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.747946 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f911511-7cb5-4eff-a4c8-628139d6ca79-tls-certs\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.848763 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.848736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-home\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.848763 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.848764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-tmp-dir\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.848970 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.848786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-dshm\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.848970 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.848804 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f911511-7cb5-4eff-a4c8-628139d6ca79-tls-certs\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.848970 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.848858 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-model-cache\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.848970 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.848901 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph2bb\" (UniqueName: \"kubernetes.io/projected/0f911511-7cb5-4eff-a4c8-628139d6ca79-kube-api-access-ph2bb\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.849300 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.849275 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-model-cache\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.849300 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.849291 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-tmp-dir\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.849480 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.849345 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-home\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.851108 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.851088 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-dshm\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.851331 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.851304 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f911511-7cb5-4eff-a4c8-628139d6ca79-tls-certs\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:41.856232 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:41.856206 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph2bb\" (UniqueName: \"kubernetes.io/projected/0f911511-7cb5-4eff-a4c8-628139d6ca79-kube-api-access-ph2bb\") pod \"auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:42.005273 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:42.005132 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:42.143093 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:42.143067 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt"] Jul 15 22:05:42.145212 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:05:42.145156 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f911511_7cb5_4eff_a4c8_628139d6ca79.slice/crio-7123618e3903a0b595a2c1465b5e35b106aef64e917ca9e32dda1b0717c5cf38 WatchSource:0}: Error finding container 7123618e3903a0b595a2c1465b5e35b106aef64e917ca9e32dda1b0717c5cf38: Status 404 returned error can't find the container with id 7123618e3903a0b595a2c1465b5e35b106aef64e917ca9e32dda1b0717c5cf38 Jul 15 22:05:43.058330 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:43.058291 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" event={"ID":"0f911511-7cb5-4eff-a4c8-628139d6ca79","Type":"ContainerStarted","Data":"0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e"} Jul 15 22:05:43.058330 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:43.058332 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" event={"ID":"0f911511-7cb5-4eff-a4c8-628139d6ca79","Type":"ContainerStarted","Data":"7123618e3903a0b595a2c1465b5e35b106aef64e917ca9e32dda1b0717c5cf38"} Jul 15 22:05:43.075611 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:43.075555 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" podStartSLOduration=2.075542066 podStartE2EDuration="2.075542066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:05:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:05:43.073051839 +0000 UTC m=+2139.653648872" watchObservedRunningTime="2026-07-15 22:05:43.075542066 +0000 UTC m=+2139.656139094" Jul 15 22:05:44.426775 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:44.426740 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:05:46.128330 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:46.128291 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:46.228798 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:46.228763 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:05:52.006000 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:52.005957 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:52.006440 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:52.006039 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:52.018511 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:52.018484 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:05:52.111420 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:05:52.111393 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:06:05.290523 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.290445 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4"] Jul 15 22:06:05.293784 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.293765 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.297472 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.297447 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"kube-root-ca.crt\"" Jul 15 22:06:05.297593 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.297488 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"openshift-service-ca.crt\"" Jul 15 22:06:05.297593 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.297447 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"default-dockercfg-l9jbm\"" Jul 15 22:06:05.297593 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.297457 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"auth-test-model-8cc1f1-kserve-self-signed-certs\"" Jul 15 22:06:05.302416 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.302393 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4"] Jul 15 22:06:05.322568 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.322533 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:06:05.335546 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.335522 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tls-certs\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.335657 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.335561 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-home\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.335657 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.335590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-dshm\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.335738 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.335658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tmp-dir\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.335738 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.335709 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qndv2\" (UniqueName: \"kubernetes.io/projected/188f3db5-8088-49b6-8b93-f6e73e5fff3d-kube-api-access-qndv2\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.335826 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.335738 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-model-cache\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.436452 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.436410 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tls-certs\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.436613 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.436461 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-home\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.436613 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.436491 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-dshm\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.436613 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.436520 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tmp-dir\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.436613 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.436550 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qndv2\" (UniqueName: \"kubernetes.io/projected/188f3db5-8088-49b6-8b93-f6e73e5fff3d-kube-api-access-qndv2\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.436613 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.436574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-model-cache\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.437047 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.437014 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-model-cache\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.437047 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.437043 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tmp-dir\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.437246 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.437066 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-home\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.438768 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.438740 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-dshm\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.438888 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.438870 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tls-certs\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.444618 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.444594 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qndv2\" (UniqueName: \"kubernetes.io/projected/188f3db5-8088-49b6-8b93-f6e73e5fff3d-kube-api-access-qndv2\") pod \"auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.603311 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.603211 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:05.732525 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:05.732339 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4"] Jul 15 22:06:05.734786 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:06:05.734760 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod188f3db5_8088_49b6_8b93_f6e73e5fff3d.slice/crio-ae7059cbcdeb5a03256648e4f0e13bcfe361a0bdbd86f00b56ec6e414805b7db WatchSource:0}: Error finding container ae7059cbcdeb5a03256648e4f0e13bcfe361a0bdbd86f00b56ec6e414805b7db: Status 404 returned error can't find the container with id ae7059cbcdeb5a03256648e4f0e13bcfe361a0bdbd86f00b56ec6e414805b7db Jul 15 22:06:06.145941 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:06.145895 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:06:06.170927 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:06.170888 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" event={"ID":"188f3db5-8088-49b6-8b93-f6e73e5fff3d","Type":"ContainerStarted","Data":"1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1"} Jul 15 22:06:06.170927 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:06.170931 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" event={"ID":"188f3db5-8088-49b6-8b93-f6e73e5fff3d","Type":"ContainerStarted","Data":"ae7059cbcdeb5a03256648e4f0e13bcfe361a0bdbd86f00b56ec6e414805b7db"} Jul 15 22:06:06.188554 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:06.188505 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" podStartSLOduration=1.18848908 podStartE2EDuration="1.18848908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:06:06.186918529 +0000 UTC m=+2162.767515559" watchObservedRunningTime="2026-07-15 22:06:06.18848908 +0000 UTC m=+2162.769086109" Jul 15 22:06:09.223118 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:09.223084 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:06:15.603661 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:15.603609 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:15.603661 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:15.603673 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:15.616328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:15.616303 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:16.230050 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:16.230021 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:06:20.429722 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:20.429689 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:06:36.825731 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:36.825562 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:06:52.933953 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:52.933915 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:06:54.582934 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:06:54.582896 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:07:09.163254 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:09.163217 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:07:09.215227 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:09.215189 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:07:12.033796 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:12.033760 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:07:30.224034 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:30.223954 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:07:35.226565 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:35.226525 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:07:35.326982 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:35.326947 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:07:43.636051 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:43.636012 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:07:58.066036 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:58.066005 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt"] Jul 15 22:07:58.080097 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:58.080052 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:07:58.980627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:58.980594 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4"] Jul 15 22:07:59.001624 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:07:59.001591 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:08:02.727304 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:02.727262 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:08:08.694184 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.694132 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch"] Jul 15 22:08:08.697280 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.697263 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.699575 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.699554 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8172c4\"/\"sub-test-model-8172c4-kserve-self-signed-certs\"" Jul 15 22:08:08.707590 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.707566 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch"] Jul 15 22:08:08.722021 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.721977 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:08:08.792394 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.792363 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-dshm\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.792533 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.792420 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tmp-dir\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.792533 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.792446 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdh4\" (UniqueName: \"kubernetes.io/projected/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-kube-api-access-tgdh4\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.792533 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.792470 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-home\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.792533 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.792490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tls-certs\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.792667 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.792605 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-model-cache\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.893634 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.893602 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-model-cache\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.893799 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.893642 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-dshm\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.893799 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.893686 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tmp-dir\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.893799 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.893703 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdh4\" (UniqueName: \"kubernetes.io/projected/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-kube-api-access-tgdh4\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.893799 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.893730 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-home\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.893799 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.893752 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tls-certs\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.894132 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.894105 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-model-cache\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.894230 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.894120 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tmp-dir\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.894230 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.894190 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-home\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.896003 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.895982 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-dshm\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.896284 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.896268 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tls-certs\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:08.901034 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:08.901008 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdh4\" (UniqueName: \"kubernetes.io/projected/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-kube-api-access-tgdh4\") pod \"sub-test-model-8172c4-kserve-7b5798f877-b6sch\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:09.006897 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:09.006812 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:09.137055 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:09.137024 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch"] Jul 15 22:08:09.138731 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:08:09.138692 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3ff3d20_ea0a_47de_82d9_0dc4e773b27a.slice/crio-f5eff8cdf7c15083d077db54846314a392bec979f6868e706d408656a2700646 WatchSource:0}: Error finding container f5eff8cdf7c15083d077db54846314a392bec979f6868e706d408656a2700646: Status 404 returned error can't find the container with id f5eff8cdf7c15083d077db54846314a392bec979f6868e706d408656a2700646 Jul 15 22:08:09.784013 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:09.783974 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" event={"ID":"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a","Type":"ContainerStarted","Data":"d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f"} Jul 15 22:08:09.784013 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:09.784024 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" event={"ID":"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a","Type":"ContainerStarted","Data":"f5eff8cdf7c15083d077db54846314a392bec979f6868e706d408656a2700646"} Jul 15 22:08:09.799395 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:09.799350 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" podStartSLOduration=1.799332394 podStartE2EDuration="1.799332394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:08:09.798200407 +0000 UTC m=+2286.378797427" watchObservedRunningTime="2026-07-15 22:08:09.799332394 +0000 UTC m=+2286.379929424" Jul 15 22:08:13.086175 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.086120 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" podUID="0f911511-7cb5-4eff-a4c8-628139d6ca79" containerName="main" containerID="cri-o://0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e" gracePeriod=45 Jul 15 22:08:13.343424 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.343352 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:08:13.436715 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.436675 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-model-cache\") pod \"0f911511-7cb5-4eff-a4c8-628139d6ca79\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " Jul 15 22:08:13.436978 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.436735 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-home\") pod \"0f911511-7cb5-4eff-a4c8-628139d6ca79\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " Jul 15 22:08:13.436978 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.436755 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f911511-7cb5-4eff-a4c8-628139d6ca79-tls-certs\") pod \"0f911511-7cb5-4eff-a4c8-628139d6ca79\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " Jul 15 22:08:13.436978 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.436784 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph2bb\" (UniqueName: \"kubernetes.io/projected/0f911511-7cb5-4eff-a4c8-628139d6ca79-kube-api-access-ph2bb\") pod \"0f911511-7cb5-4eff-a4c8-628139d6ca79\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " Jul 15 22:08:13.436978 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.436814 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-dshm\") pod \"0f911511-7cb5-4eff-a4c8-628139d6ca79\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " Jul 15 22:08:13.436978 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.436871 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-tmp-dir\") pod \"0f911511-7cb5-4eff-a4c8-628139d6ca79\" (UID: \"0f911511-7cb5-4eff-a4c8-628139d6ca79\") " Jul 15 22:08:13.437294 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.437007 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f911511-7cb5-4eff-a4c8-628139d6ca79" (UID: "0f911511-7cb5-4eff-a4c8-628139d6ca79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:13.437294 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.437024 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-home" (OuterVolumeSpecName: "home") pod "0f911511-7cb5-4eff-a4c8-628139d6ca79" (UID: "0f911511-7cb5-4eff-a4c8-628139d6ca79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:13.437294 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.437101 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:13.437294 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.437119 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:13.437294 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.437205 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f911511-7cb5-4eff-a4c8-628139d6ca79" (UID: "0f911511-7cb5-4eff-a4c8-628139d6ca79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:13.439192 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.439137 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-dshm" (OuterVolumeSpecName: "dshm") pod "0f911511-7cb5-4eff-a4c8-628139d6ca79" (UID: "0f911511-7cb5-4eff-a4c8-628139d6ca79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:13.439295 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.439192 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f911511-7cb5-4eff-a4c8-628139d6ca79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f911511-7cb5-4eff-a4c8-628139d6ca79" (UID: "0f911511-7cb5-4eff-a4c8-628139d6ca79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:08:13.439295 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.439229 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f911511-7cb5-4eff-a4c8-628139d6ca79-kube-api-access-ph2bb" (OuterVolumeSpecName: "kube-api-access-ph2bb") pod "0f911511-7cb5-4eff-a4c8-628139d6ca79" (UID: "0f911511-7cb5-4eff-a4c8-628139d6ca79"). InnerVolumeSpecName "kube-api-access-ph2bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:08:13.537912 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.537872 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f911511-7cb5-4eff-a4c8-628139d6ca79-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:13.537912 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.537906 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph2bb\" (UniqueName: \"kubernetes.io/projected/0f911511-7cb5-4eff-a4c8-628139d6ca79-kube-api-access-ph2bb\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:13.537912 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.537915 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:13.538151 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.537925 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f911511-7cb5-4eff-a4c8-628139d6ca79-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:13.806338 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.806302 2572 generic.go:356] "Generic (PLEG): container finished" podID="0f911511-7cb5-4eff-a4c8-628139d6ca79" containerID="0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e" exitCode=0 Jul 15 22:08:13.806519 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.806350 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" Jul 15 22:08:13.806519 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.806388 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" event={"ID":"0f911511-7cb5-4eff-a4c8-628139d6ca79","Type":"ContainerDied","Data":"0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e"} Jul 15 22:08:13.806519 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.806438 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt" event={"ID":"0f911511-7cb5-4eff-a4c8-628139d6ca79","Type":"ContainerDied","Data":"7123618e3903a0b595a2c1465b5e35b106aef64e917ca9e32dda1b0717c5cf38"} Jul 15 22:08:13.806519 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.806460 2572 scope.go:117] "RemoveContainer" containerID="0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e" Jul 15 22:08:13.815863 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.815846 2572 scope.go:117] "RemoveContainer" containerID="0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e" Jul 15 22:08:13.816116 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:08:13.816097 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e\": container with ID starting with 0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e not found: ID does not exist" containerID="0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e" Jul 15 22:08:13.816197 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.816129 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e"} err="failed to get container status \"0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e\": rpc error: code = NotFound desc = could not find container \"0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e\": container with ID starting with 0f3158ba7ee7d32cec819f46e6d631ead4b3a7fe3d9d0ed7f793a9baef5df17e not found: ID does not exist" Jul 15 22:08:13.827712 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.827692 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt"] Jul 15 22:08:13.829537 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:13.829507 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/auth-test-model-8172c4-kserve-84fc9f59c9-gh2qt"] Jul 15 22:08:14.000753 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.000716 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" podUID="188f3db5-8088-49b6-8b93-f6e73e5fff3d" containerName="main" containerID="cri-o://1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1" gracePeriod=45 Jul 15 22:08:14.063690 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.063619 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f911511-7cb5-4eff-a4c8-628139d6ca79" path="/var/lib/kubelet/pods/0f911511-7cb5-4eff-a4c8-628139d6ca79/volumes" Jul 15 22:08:14.246959 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.246937 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:08:14.343571 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343488 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-home\") pod \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " Jul 15 22:08:14.343571 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343538 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-model-cache\") pod \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " Jul 15 22:08:14.343817 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343574 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qndv2\" (UniqueName: \"kubernetes.io/projected/188f3db5-8088-49b6-8b93-f6e73e5fff3d-kube-api-access-qndv2\") pod \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " Jul 15 22:08:14.343817 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343619 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-dshm\") pod \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " Jul 15 22:08:14.343817 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343646 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tls-certs\") pod \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " Jul 15 22:08:14.343817 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343668 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tmp-dir\") pod \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\" (UID: \"188f3db5-8088-49b6-8b93-f6e73e5fff3d\") " Jul 15 22:08:14.343817 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343799 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-home" (OuterVolumeSpecName: "home") pod "188f3db5-8088-49b6-8b93-f6e73e5fff3d" (UID: "188f3db5-8088-49b6-8b93-f6e73e5fff3d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:14.344065 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343917 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-model-cache" (OuterVolumeSpecName: "model-cache") pod "188f3db5-8088-49b6-8b93-f6e73e5fff3d" (UID: "188f3db5-8088-49b6-8b93-f6e73e5fff3d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:14.344065 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.343974 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "188f3db5-8088-49b6-8b93-f6e73e5fff3d" (UID: "188f3db5-8088-49b6-8b93-f6e73e5fff3d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:14.344065 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.344055 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:14.344179 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.344084 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:14.344179 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.344101 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:14.346008 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.345984 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "188f3db5-8088-49b6-8b93-f6e73e5fff3d" (UID: "188f3db5-8088-49b6-8b93-f6e73e5fff3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:08:14.346092 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.345987 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188f3db5-8088-49b6-8b93-f6e73e5fff3d-kube-api-access-qndv2" (OuterVolumeSpecName: "kube-api-access-qndv2") pod "188f3db5-8088-49b6-8b93-f6e73e5fff3d" (UID: "188f3db5-8088-49b6-8b93-f6e73e5fff3d"). InnerVolumeSpecName "kube-api-access-qndv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:08:14.346092 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.346055 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-dshm" (OuterVolumeSpecName: "dshm") pod "188f3db5-8088-49b6-8b93-f6e73e5fff3d" (UID: "188f3db5-8088-49b6-8b93-f6e73e5fff3d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:08:14.445053 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.445018 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qndv2\" (UniqueName: \"kubernetes.io/projected/188f3db5-8088-49b6-8b93-f6e73e5fff3d-kube-api-access-qndv2\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:14.445053 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.445048 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/188f3db5-8088-49b6-8b93-f6e73e5fff3d-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:14.445053 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.445059 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/188f3db5-8088-49b6-8b93-f6e73e5fff3d-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:08:14.812399 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.812360 2572 generic.go:356] "Generic (PLEG): container finished" podID="188f3db5-8088-49b6-8b93-f6e73e5fff3d" containerID="1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1" exitCode=0 Jul 15 22:08:14.812574 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.812413 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" Jul 15 22:08:14.812574 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.812440 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" event={"ID":"188f3db5-8088-49b6-8b93-f6e73e5fff3d","Type":"ContainerDied","Data":"1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1"} Jul 15 22:08:14.812574 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.812485 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4" event={"ID":"188f3db5-8088-49b6-8b93-f6e73e5fff3d","Type":"ContainerDied","Data":"ae7059cbcdeb5a03256648e4f0e13bcfe361a0bdbd86f00b56ec6e414805b7db"} Jul 15 22:08:14.812574 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.812507 2572 scope.go:117] "RemoveContainer" containerID="1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1" Jul 15 22:08:14.821598 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.821580 2572 scope.go:117] "RemoveContainer" containerID="1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1" Jul 15 22:08:14.821827 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:08:14.821808 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1\": container with ID starting with 1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1 not found: ID does not exist" containerID="1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1" Jul 15 22:08:14.821871 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.821836 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1"} err="failed to get container status \"1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1\": rpc error: code = NotFound desc = could not find container \"1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1\": container with ID starting with 1ac04e3a1b5f79bda89dd642ef52f48163f549f4e0a7f94a6a52974c8b0afab1 not found: ID does not exist" Jul 15 22:08:14.833923 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.833902 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4"] Jul 15 22:08:14.836790 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:14.836767 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/auth-test-model-8cc1f1-kserve-7f55bdc7c5-4qqh4"] Jul 15 22:08:16.063244 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:16.063203 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188f3db5-8088-49b6-8b93-f6e73e5fff3d" path="/var/lib/kubelet/pods/188f3db5-8088-49b6-8b93-f6e73e5fff3d/volumes" Jul 15 22:08:19.007299 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:19.007262 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:19.007699 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:19.007321 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:19.019772 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:19.019738 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:19.524330 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:19.524294 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:08:19.850132 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:19.850049 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:08:22.377150 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377109 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f"] Jul 15 22:08:22.377622 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377444 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188f3db5-8088-49b6-8b93-f6e73e5fff3d" containerName="main" Jul 15 22:08:22.377622 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377457 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="188f3db5-8088-49b6-8b93-f6e73e5fff3d" containerName="main" Jul 15 22:08:22.377622 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377473 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f911511-7cb5-4eff-a4c8-628139d6ca79" containerName="main" Jul 15 22:08:22.377622 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377478 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f911511-7cb5-4eff-a4c8-628139d6ca79" containerName="main" Jul 15 22:08:22.377622 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377528 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f911511-7cb5-4eff-a4c8-628139d6ca79" containerName="main" Jul 15 22:08:22.377622 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.377537 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="188f3db5-8088-49b6-8b93-f6e73e5fff3d" containerName="main" Jul 15 22:08:22.380272 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.380251 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.383264 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.383242 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"kube-root-ca.crt\"" Jul 15 22:08:22.383407 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.383331 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"sub-test-model-8cc1f1-kserve-self-signed-certs\"" Jul 15 22:08:22.383407 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.383336 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"default-dockercfg-l9jbm\"" Jul 15 22:08:22.383407 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.383362 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"openshift-service-ca.crt\"" Jul 15 22:08:22.390131 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.390107 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f"] Jul 15 22:08:22.407145 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.407122 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:08:22.414242 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.414221 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-dshm\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.414336 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.414249 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-model-cache\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.414336 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.414268 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm569\" (UniqueName: \"kubernetes.io/projected/835cb6fb-67a9-45b2-95c2-16c2dacb7166-kube-api-access-vm569\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.414336 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.414287 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tls-certs\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.414454 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.414373 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-home\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.414454 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.414399 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tmp-dir\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.515735 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.515700 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-home\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.515919 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.515742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tmp-dir\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.515919 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.515815 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-dshm\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.515919 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.515838 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-model-cache\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.515919 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.515861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm569\" (UniqueName: \"kubernetes.io/projected/835cb6fb-67a9-45b2-95c2-16c2dacb7166-kube-api-access-vm569\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.515919 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.515886 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tls-certs\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.516229 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.516205 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-home\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.516303 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.516229 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tmp-dir\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.516303 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.516258 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-model-cache\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.518146 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.518124 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-dshm\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.518468 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.518448 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tls-certs\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.523145 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.523122 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm569\" (UniqueName: \"kubernetes.io/projected/835cb6fb-67a9-45b2-95c2-16c2dacb7166-kube-api-access-vm569\") pod \"sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.689717 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.689625 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:22.820143 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.820120 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f"] Jul 15 22:08:22.822414 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:08:22.822388 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod835cb6fb_67a9_45b2_95c2_16c2dacb7166.slice/crio-cd88405940c4c5e98a0961867a70d55b1e64e461af089d72c6e09325347052a9 WatchSource:0}: Error finding container cd88405940c4c5e98a0961867a70d55b1e64e461af089d72c6e09325347052a9: Status 404 returned error can't find the container with id cd88405940c4c5e98a0961867a70d55b1e64e461af089d72c6e09325347052a9 Jul 15 22:08:22.856272 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:22.856229 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" event={"ID":"835cb6fb-67a9-45b2-95c2-16c2dacb7166","Type":"ContainerStarted","Data":"cd88405940c4c5e98a0961867a70d55b1e64e461af089d72c6e09325347052a9"} Jul 15 22:08:23.863159 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:23.863126 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" event={"ID":"835cb6fb-67a9-45b2-95c2-16c2dacb7166","Type":"ContainerStarted","Data":"5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0"} Jul 15 22:08:23.878821 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:23.878773 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" podStartSLOduration=1.87875768 podStartE2EDuration="1.87875768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:08:23.877106288 +0000 UTC m=+2300.457703316" watchObservedRunningTime="2026-07-15 22:08:23.87875768 +0000 UTC m=+2300.459354709" Jul 15 22:08:32.689876 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:32.689839 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:32.690315 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:32.690036 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:32.702295 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:32.702273 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:32.925577 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:32.925548 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:08:37.827570 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:37.827537 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:08:51.733756 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:08:51.733720 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:09:09.264521 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:09.264485 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch"] Jul 15 22:09:09.285670 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:09.285616 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:09:10.092503 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:10.092474 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f"] Jul 15 22:09:10.112260 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:10.112227 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:09:11.829773 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:11.829736 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:09:12.587107 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.587070 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84"] Jul 15 22:09:12.590297 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.590277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.592521 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.592504 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-8172c4\"/\"rate-test-model-8172c4-kserve-self-signed-certs\"" Jul 15 22:09:12.599114 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.599089 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84"] Jul 15 22:09:12.623751 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.623719 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:09:12.640100 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.640072 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmvgc\" (UniqueName: \"kubernetes.io/projected/fbc303eb-532b-4219-8b16-79b23ed5a1a0-kube-api-access-gmvgc\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.640250 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.640119 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-model-cache\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.640250 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.640204 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-dshm\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.640250 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.640235 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tls-certs\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.640359 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.640260 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-home\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.640359 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.640304 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tmp-dir\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.740962 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.740924 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-home\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.740962 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.740971 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tmp-dir\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741249 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741012 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmvgc\" (UniqueName: \"kubernetes.io/projected/fbc303eb-532b-4219-8b16-79b23ed5a1a0-kube-api-access-gmvgc\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741249 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741047 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-model-cache\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741249 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-dshm\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741249 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741118 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tls-certs\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741471 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741445 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-home\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741530 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741481 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tmp-dir\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.741585 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.741536 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-model-cache\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.743385 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.743363 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-dshm\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.743772 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.743747 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tls-certs\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.748792 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.748761 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmvgc\" (UniqueName: \"kubernetes.io/projected/fbc303eb-532b-4219-8b16-79b23ed5a1a0-kube-api-access-gmvgc\") pod \"rate-test-model-8172c4-kserve-6f4c46956c-rbt84\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:12.900730 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:12.900643 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:13.031008 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:13.030983 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84"] Jul 15 22:09:13.033203 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:09:13.033159 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbc303eb_532b_4219_8b16_79b23ed5a1a0.slice/crio-a3d65badfeb982ff195948d343a3ea74b1fcbb7d314794660783594a5b3d8df7 WatchSource:0}: Error finding container a3d65badfeb982ff195948d343a3ea74b1fcbb7d314794660783594a5b3d8df7: Status 404 returned error can't find the container with id a3d65badfeb982ff195948d343a3ea74b1fcbb7d314794660783594a5b3d8df7 Jul 15 22:09:13.108508 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:13.108478 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" event={"ID":"fbc303eb-532b-4219-8b16-79b23ed5a1a0","Type":"ContainerStarted","Data":"f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1"} Jul 15 22:09:13.108637 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:13.108518 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" event={"ID":"fbc303eb-532b-4219-8b16-79b23ed5a1a0","Type":"ContainerStarted","Data":"a3d65badfeb982ff195948d343a3ea74b1fcbb7d314794660783594a5b3d8df7"} Jul 15 22:09:13.125436 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:13.125390 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" podStartSLOduration=1.125375608 podStartE2EDuration="1.125375608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:09:13.12327217 +0000 UTC m=+2349.703869199" watchObservedRunningTime="2026-07-15 22:09:13.125375608 +0000 UTC m=+2349.705972636" Jul 15 22:09:22.901482 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:22.901446 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:22.901953 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:22.901495 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:22.919198 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:22.919154 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:23.174486 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:23.174407 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:09:24.285834 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.285793 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" podUID="a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" containerName="main" containerID="cri-o://d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f" gracePeriod=45 Jul 15 22:09:24.537911 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.537842 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:09:24.641696 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.641655 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-dshm\") pod \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " Jul 15 22:09:24.641888 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.641704 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-model-cache\") pod \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " Jul 15 22:09:24.641888 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.641752 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgdh4\" (UniqueName: \"kubernetes.io/projected/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-kube-api-access-tgdh4\") pod \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " Jul 15 22:09:24.641888 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.641772 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-home\") pod \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " Jul 15 22:09:24.641888 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.641816 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tmp-dir\") pod \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " Jul 15 22:09:24.641888 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.641852 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tls-certs\") pod \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\" (UID: \"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a\") " Jul 15 22:09:24.642151 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.642009 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" (UID: "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:24.642151 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.642102 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" (UID: "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:24.642151 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.642067 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-home" (OuterVolumeSpecName: "home") pod "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" (UID: "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:24.642151 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.642115 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:24.643957 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.643931 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-dshm" (OuterVolumeSpecName: "dshm") pod "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" (UID: "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:24.643957 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.643944 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" (UID: "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:09:24.644117 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.643999 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-kube-api-access-tgdh4" (OuterVolumeSpecName: "kube-api-access-tgdh4") pod "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" (UID: "a3ff3d20-ea0a-47de-82d9-0dc4e773b27a"). InnerVolumeSpecName "kube-api-access-tgdh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:09:24.742828 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.742790 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdh4\" (UniqueName: \"kubernetes.io/projected/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-kube-api-access-tgdh4\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:24.742828 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.742822 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:24.742828 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.742832 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:24.743050 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.742841 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:24.743050 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:24.742849 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:25.112995 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.112935 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" podUID="835cb6fb-67a9-45b2-95c2-16c2dacb7166" containerName="main" containerID="cri-o://5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0" gracePeriod=45 Jul 15 22:09:25.174284 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.174246 2572 generic.go:356] "Generic (PLEG): container finished" podID="a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" containerID="d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f" exitCode=0 Jul 15 22:09:25.174284 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.174286 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" event={"ID":"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a","Type":"ContainerDied","Data":"d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f"} Jul 15 22:09:25.174512 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.174312 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" Jul 15 22:09:25.174512 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.174322 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch" event={"ID":"a3ff3d20-ea0a-47de-82d9-0dc4e773b27a","Type":"ContainerDied","Data":"f5eff8cdf7c15083d077db54846314a392bec979f6868e706d408656a2700646"} Jul 15 22:09:25.174512 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.174339 2572 scope.go:117] "RemoveContainer" containerID="d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f" Jul 15 22:09:25.193549 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.193525 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch"] Jul 15 22:09:25.197218 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.197194 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/sub-test-model-8172c4-kserve-7b5798f877-b6sch"] Jul 15 22:09:25.223297 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.223271 2572 scope.go:117] "RemoveContainer" containerID="d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f" Jul 15 22:09:25.223636 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:09:25.223612 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f\": container with ID starting with d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f not found: ID does not exist" containerID="d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f" Jul 15 22:09:25.223715 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.223650 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f"} err="failed to get container status \"d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f\": rpc error: code = NotFound desc = could not find container \"d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f\": container with ID starting with d8cad19577b8292cd8fd5fe03331decd39b1027ea3cb832f16b6b067ce50c00f not found: ID does not exist" Jul 15 22:09:25.362284 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.362257 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:09:25.449023 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.448989 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm569\" (UniqueName: \"kubernetes.io/projected/835cb6fb-67a9-45b2-95c2-16c2dacb7166-kube-api-access-vm569\") pod \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " Jul 15 22:09:25.449233 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449038 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tls-certs\") pod \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " Jul 15 22:09:25.449233 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449103 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tmp-dir\") pod \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " Jul 15 22:09:25.449233 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449151 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-dshm\") pod \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " Jul 15 22:09:25.449233 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449205 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-model-cache\") pod \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " Jul 15 22:09:25.449447 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449239 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-home\") pod \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\" (UID: \"835cb6fb-67a9-45b2-95c2-16c2dacb7166\") " Jul 15 22:09:25.449502 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449454 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "835cb6fb-67a9-45b2-95c2-16c2dacb7166" (UID: "835cb6fb-67a9-45b2-95c2-16c2dacb7166"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:25.449542 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449496 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-model-cache" (OuterVolumeSpecName: "model-cache") pod "835cb6fb-67a9-45b2-95c2-16c2dacb7166" (UID: "835cb6fb-67a9-45b2-95c2-16c2dacb7166"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:25.449586 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.449556 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-home" (OuterVolumeSpecName: "home") pod "835cb6fb-67a9-45b2-95c2-16c2dacb7166" (UID: "835cb6fb-67a9-45b2-95c2-16c2dacb7166"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:25.451281 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.451261 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "835cb6fb-67a9-45b2-95c2-16c2dacb7166" (UID: "835cb6fb-67a9-45b2-95c2-16c2dacb7166"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:09:25.451281 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.451273 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835cb6fb-67a9-45b2-95c2-16c2dacb7166-kube-api-access-vm569" (OuterVolumeSpecName: "kube-api-access-vm569") pod "835cb6fb-67a9-45b2-95c2-16c2dacb7166" (UID: "835cb6fb-67a9-45b2-95c2-16c2dacb7166"). InnerVolumeSpecName "kube-api-access-vm569". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:09:25.451406 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.451311 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-dshm" (OuterVolumeSpecName: "dshm") pod "835cb6fb-67a9-45b2-95c2-16c2dacb7166" (UID: "835cb6fb-67a9-45b2-95c2-16c2dacb7166"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:09:25.550173 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.550142 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:25.550328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.550183 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm569\" (UniqueName: \"kubernetes.io/projected/835cb6fb-67a9-45b2-95c2-16c2dacb7166-kube-api-access-vm569\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:25.550328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.550195 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:25.550328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.550204 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:25.550328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.550211 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:25.550328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:25.550218 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/835cb6fb-67a9-45b2-95c2-16c2dacb7166-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:09:26.063134 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.063100 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" path="/var/lib/kubelet/pods/a3ff3d20-ea0a-47de-82d9-0dc4e773b27a/volumes" Jul 15 22:09:26.181426 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.181391 2572 generic.go:356] "Generic (PLEG): container finished" podID="835cb6fb-67a9-45b2-95c2-16c2dacb7166" containerID="5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0" exitCode=0 Jul 15 22:09:26.181584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.181435 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" Jul 15 22:09:26.181584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.181451 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" event={"ID":"835cb6fb-67a9-45b2-95c2-16c2dacb7166","Type":"ContainerDied","Data":"5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0"} Jul 15 22:09:26.181584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.181487 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f" event={"ID":"835cb6fb-67a9-45b2-95c2-16c2dacb7166","Type":"ContainerDied","Data":"cd88405940c4c5e98a0961867a70d55b1e64e461af089d72c6e09325347052a9"} Jul 15 22:09:26.181584 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.181509 2572 scope.go:117] "RemoveContainer" containerID="5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0" Jul 15 22:09:26.190461 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.190439 2572 scope.go:117] "RemoveContainer" containerID="5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0" Jul 15 22:09:26.190711 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:09:26.190692 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0\": container with ID starting with 5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0 not found: ID does not exist" containerID="5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0" Jul 15 22:09:26.190782 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.190721 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0"} err="failed to get container status \"5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0\": rpc error: code = NotFound desc = could not find container \"5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0\": container with ID starting with 5871f2323a98b91dc731bca77c9e0827e900e85cba4e915e5a7a17bf0763f6b0 not found: ID does not exist" Jul 15 22:09:26.196896 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.196873 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f"] Jul 15 22:09:26.199989 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:26.199968 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/sub-test-model-8cc1f1-kserve-5589bddf9-6rj5f"] Jul 15 22:09:28.061676 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:28.061643 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="835cb6fb-67a9-45b2-95c2-16c2dacb7166" path="/var/lib/kubelet/pods/835cb6fb-67a9-45b2-95c2-16c2dacb7166/volumes" Jul 15 22:09:29.423854 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:29.423823 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:09:36.085206 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085174 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc"] Jul 15 22:09:36.085627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085492 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="835cb6fb-67a9-45b2-95c2-16c2dacb7166" containerName="main" Jul 15 22:09:36.085627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085505 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="835cb6fb-67a9-45b2-95c2-16c2dacb7166" containerName="main" Jul 15 22:09:36.085627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085521 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" containerName="main" Jul 15 22:09:36.085627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085526 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" containerName="main" Jul 15 22:09:36.085627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085586 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="835cb6fb-67a9-45b2-95c2-16c2dacb7166" containerName="main" Jul 15 22:09:36.085627 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.085595 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3ff3d20-ea0a-47de-82d9-0dc4e773b27a" containerName="main" Jul 15 22:09:36.090128 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.090107 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.092630 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.092604 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"openshift-service-ca.crt\"" Jul 15 22:09:36.093427 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.093379 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"rate-test-model-8cc1f1-kserve-self-signed-certs\"" Jul 15 22:09:36.093532 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.093442 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"kube-root-ca.crt\"" Jul 15 22:09:36.093532 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.093480 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8cc1f1\"/\"default-dockercfg-l9jbm\"" Jul 15 22:09:36.099028 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.098994 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc"] Jul 15 22:09:36.115595 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.115559 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:09:36.141348 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.141313 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-model-cache\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.141518 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.141387 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-home\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.141518 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.141413 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tls-certs\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.141518 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.141480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-dshm\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.141518 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.141505 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpbkc\" (UniqueName: \"kubernetes.io/projected/2a6ec3b3-9dcc-4a86-8448-84163a525b26-kube-api-access-jpbkc\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.141683 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.141540 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tmp-dir\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.242753 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.242717 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-home\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.242753 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.242750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tls-certs\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243009 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.242786 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-dshm\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243009 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.242897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpbkc\" (UniqueName: \"kubernetes.io/projected/2a6ec3b3-9dcc-4a86-8448-84163a525b26-kube-api-access-jpbkc\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243009 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.242944 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tmp-dir\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243009 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.242999 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-model-cache\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243260 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.243150 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-home\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243359 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.243340 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-model-cache\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.243399 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.243342 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tmp-dir\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.245149 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.245132 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-dshm\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.245436 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.245420 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tls-certs\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.250514 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.250489 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpbkc\" (UniqueName: \"kubernetes.io/projected/2a6ec3b3-9dcc-4a86-8448-84163a525b26-kube-api-access-jpbkc\") pod \"rate-test-model-8cc1f1-kserve-7679878b7f-94lcc\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.401497 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.401408 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:36.530570 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:36.530544 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc"] Jul 15 22:09:36.532789 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:09:36.532756 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a6ec3b3_9dcc_4a86_8448_84163a525b26.slice/crio-99342cd5832c3a1d3269757152fa1b171188b7cb13bfe40f5324035741718fba WatchSource:0}: Error finding container 99342cd5832c3a1d3269757152fa1b171188b7cb13bfe40f5324035741718fba: Status 404 returned error can't find the container with id 99342cd5832c3a1d3269757152fa1b171188b7cb13bfe40f5324035741718fba Jul 15 22:09:37.236484 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:37.236449 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" event={"ID":"2a6ec3b3-9dcc-4a86-8448-84163a525b26","Type":"ContainerStarted","Data":"ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8"} Jul 15 22:09:37.236484 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:37.236492 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" event={"ID":"2a6ec3b3-9dcc-4a86-8448-84163a525b26","Type":"ContainerStarted","Data":"99342cd5832c3a1d3269757152fa1b171188b7cb13bfe40f5324035741718fba"} Jul 15 22:09:37.253705 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:37.253649 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" podStartSLOduration=1.253631502 podStartE2EDuration="1.253631502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:09:37.253257891 +0000 UTC m=+2373.833854933" watchObservedRunningTime="2026-07-15 22:09:37.253631502 +0000 UTC m=+2373.834228532" Jul 15 22:09:42.430803 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:42.430768 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:09:46.401836 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:46.401797 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:46.401836 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:46.401847 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:46.414848 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:46.414827 2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:47.296533 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:47.296502 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:09:57.735261 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:09:57.735224 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:10:04.057813 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:04.057785 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 22:10:04.067071 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:04.067046 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-lnzxm_9c893c1a-a131-4b5e-ba1d-74e25f8eb7d1/ovn-acl-logging/0.log" Jul 15 22:10:18.247581 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:18.247541 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:10:24.077580 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:24.077546 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84"] Jul 15 22:10:24.095945 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:24.095909 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:24.973903 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:24.973870 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc"] Jul 15 22:10:24.992634 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:24.992598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:25.937933 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:25.937853 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7869-140337491684528"] Jul 15 22:10:25.940785 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:25.940768 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:25.942727 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:25.942705 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cdkg4\"" Jul 15 22:10:25.947190 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:25.947147 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491684528"] Jul 15 22:10:26.073632 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.073597 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75l8\" (UniqueName: \"kubernetes.io/projected/41ebbe8e-f2cb-4505-a27f-acfa0b684841-kube-api-access-m75l8\") pod \"test-curl-7869-140337491684528\" (UID: \"41ebbe8e-f2cb-4505-a27f-acfa0b684841\") " pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:26.174676 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.174640 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75l8\" (UniqueName: \"kubernetes.io/projected/41ebbe8e-f2cb-4505-a27f-acfa0b684841-kube-api-access-m75l8\") pod \"test-curl-7869-140337491684528\" (UID: \"41ebbe8e-f2cb-4505-a27f-acfa0b684841\") " pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:26.182223 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.182193 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75l8\" (UniqueName: \"kubernetes.io/projected/41ebbe8e-f2cb-4505-a27f-acfa0b684841-kube-api-access-m75l8\") pod \"test-curl-7869-140337491684528\" (UID: \"41ebbe8e-f2cb-4505-a27f-acfa0b684841\") " pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:26.251521 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.251487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:26.377925 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.377883 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491684528"] Jul 15 22:10:26.380146 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:10:26.380121 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41ebbe8e_f2cb_4505_a27f_acfa0b684841.slice/crio-bbc19a18329c1e944b1ea630f7dae6afebc8dc4be5ac982fb9494e619cadb8c6 WatchSource:0}: Error finding container bbc19a18329c1e944b1ea630f7dae6afebc8dc4be5ac982fb9494e619cadb8c6: Status 404 returned error can't find the container with id bbc19a18329c1e944b1ea630f7dae6afebc8dc4be5ac982fb9494e619cadb8c6 Jul 15 22:10:26.381386 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.381366 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 22:10:26.475420 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:26.475390 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491684528" event={"ID":"41ebbe8e-f2cb-4505-a27f-acfa0b684841","Type":"ContainerStarted","Data":"bbc19a18329c1e944b1ea630f7dae6afebc8dc4be5ac982fb9494e619cadb8c6"} Jul 15 22:10:27.481689 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:27.481655 2572 generic.go:356] "Generic (PLEG): container finished" podID="41ebbe8e-f2cb-4505-a27f-acfa0b684841" containerID="988cf698bfb96a4600c25da58282110f23a7791105182098f779d94344c4811e" exitCode=0 Jul 15 22:10:27.482064 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:27.481715 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491684528" event={"ID":"41ebbe8e-f2cb-4505-a27f-acfa0b684841","Type":"ContainerDied","Data":"988cf698bfb96a4600c25da58282110f23a7791105182098f779d94344c4811e"} Jul 15 22:10:28.613334 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.613306 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:28.632189 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.632120 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7869-140337491684528_41ebbe8e-f2cb-4505-a27f-acfa0b684841/test-curl-7869-140337491684528/0.log" Jul 15 22:10:28.654244 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.654215 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7869-140337491684528"] Jul 15 22:10:28.656375 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.656353 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7869-140337491684528"] Jul 15 22:10:28.697243 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.697219 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m75l8\" (UniqueName: \"kubernetes.io/projected/41ebbe8e-f2cb-4505-a27f-acfa0b684841-kube-api-access-m75l8\") pod \"41ebbe8e-f2cb-4505-a27f-acfa0b684841\" (UID: \"41ebbe8e-f2cb-4505-a27f-acfa0b684841\") " Jul 15 22:10:28.699367 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.699336 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ebbe8e-f2cb-4505-a27f-acfa0b684841-kube-api-access-m75l8" (OuterVolumeSpecName: "kube-api-access-m75l8") pod "41ebbe8e-f2cb-4505-a27f-acfa0b684841" (UID: "41ebbe8e-f2cb-4505-a27f-acfa0b684841"). InnerVolumeSpecName "kube-api-access-m75l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:28.782780 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.782689 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7869-140337491683760"] Jul 15 22:10:28.783102 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.783084 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41ebbe8e-f2cb-4505-a27f-acfa0b684841" containerName="test-curl-7869-140337491684528" Jul 15 22:10:28.783216 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.783106 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="41ebbe8e-f2cb-4505-a27f-acfa0b684841" containerName="test-curl-7869-140337491684528" Jul 15 22:10:28.783281 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.783240 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="41ebbe8e-f2cb-4505-a27f-acfa0b684841" containerName="test-curl-7869-140337491684528" Jul 15 22:10:28.786190 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.786153 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:28.790713 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.790689 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491683760"] Jul 15 22:10:28.798132 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.798113 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m75l8\" (UniqueName: \"kubernetes.io/projected/41ebbe8e-f2cb-4505-a27f-acfa0b684841-kube-api-access-m75l8\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:28.899396 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:28.899366 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd62w\" (UniqueName: \"kubernetes.io/projected/af4119c8-f53e-430e-af20-cf475397e2c7-kube-api-access-bd62w\") pod \"test-curl-7869-140337491683760\" (UID: \"af4119c8-f53e-430e-af20-cf475397e2c7\") " pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:29.000782 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.000750 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd62w\" (UniqueName: \"kubernetes.io/projected/af4119c8-f53e-430e-af20-cf475397e2c7-kube-api-access-bd62w\") pod \"test-curl-7869-140337491683760\" (UID: \"af4119c8-f53e-430e-af20-cf475397e2c7\") " pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:29.008582 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.008556 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd62w\" (UniqueName: \"kubernetes.io/projected/af4119c8-f53e-430e-af20-cf475397e2c7-kube-api-access-bd62w\") pod \"test-curl-7869-140337491683760\" (UID: \"af4119c8-f53e-430e-af20-cf475397e2c7\") " pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:29.096937 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.096853 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:29.221473 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.221399 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491683760"] Jul 15 22:10:29.224096 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:10:29.224062 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf4119c8_f53e_430e_af20_cf475397e2c7.slice/crio-9e65c1676c1c0f71db56b4c3f080be3919e885cfc192ba631d023ad677b90684 WatchSource:0}: Error finding container 9e65c1676c1c0f71db56b4c3f080be3919e885cfc192ba631d023ad677b90684: Status 404 returned error can't find the container with id 9e65c1676c1c0f71db56b4c3f080be3919e885cfc192ba631d023ad677b90684 Jul 15 22:10:29.492263 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.492230 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491683760" event={"ID":"af4119c8-f53e-430e-af20-cf475397e2c7","Type":"ContainerStarted","Data":"9e65c1676c1c0f71db56b4c3f080be3919e885cfc192ba631d023ad677b90684"} Jul 15 22:10:29.493544 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.493519 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc19a18329c1e944b1ea630f7dae6afebc8dc4be5ac982fb9494e619cadb8c6" Jul 15 22:10:29.493674 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:29.493548 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:29.503674 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:10:29.503645 2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7869-140337491684528\" is forbidden: User \"system:node:ip-10-0-135-178.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-178.ec2.internal' and this object" podUID="41ebbe8e-f2cb-4505-a27f-acfa0b684841" pod="opendatahub/test-curl-7869-140337491684528" Jul 15 22:10:30.062015 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:30.061980 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41ebbe8e-f2cb-4505-a27f-acfa0b684841" path="/var/lib/kubelet/pods/41ebbe8e-f2cb-4505-a27f-acfa0b684841/volumes" Jul 15 22:10:30.500237 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:30.500202 2572 generic.go:356] "Generic (PLEG): container finished" podID="af4119c8-f53e-430e-af20-cf475397e2c7" containerID="bb15c5b8e999436c5c3105736c41d9d8d63ba689d7a4ff3a9479be40acdb12d6" exitCode=0 Jul 15 22:10:30.500419 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:30.500300 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491683760" event={"ID":"af4119c8-f53e-430e-af20-cf475397e2c7","Type":"ContainerDied","Data":"bb15c5b8e999436c5c3105736c41d9d8d63ba689d7a4ff3a9479be40acdb12d6"} Jul 15 22:10:31.635322 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.635289 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:31.653243 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.653211 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7869-140337491683760_af4119c8-f53e-430e-af20-cf475397e2c7/test-curl-7869-140337491683760/0.log" Jul 15 22:10:31.682780 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.682746 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7869-140337491683760"] Jul 15 22:10:31.687353 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.687329 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7869-140337491683760"] Jul 15 22:10:31.723819 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.723779 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd62w\" (UniqueName: \"kubernetes.io/projected/af4119c8-f53e-430e-af20-cf475397e2c7-kube-api-access-bd62w\") pod \"af4119c8-f53e-430e-af20-cf475397e2c7\" (UID: \"af4119c8-f53e-430e-af20-cf475397e2c7\") " Jul 15 22:10:31.725943 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.725922 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af4119c8-f53e-430e-af20-cf475397e2c7-kube-api-access-bd62w" (OuterVolumeSpecName: "kube-api-access-bd62w") pod "af4119c8-f53e-430e-af20-cf475397e2c7" (UID: "af4119c8-f53e-430e-af20-cf475397e2c7"). InnerVolumeSpecName "kube-api-access-bd62w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:31.825290 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.825213 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd62w\" (UniqueName: \"kubernetes.io/projected/af4119c8-f53e-430e-af20-cf475397e2c7-kube-api-access-bd62w\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:31.846646 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.846621 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:31.846932 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.846919 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af4119c8-f53e-430e-af20-cf475397e2c7" containerName="test-curl-7869-140337491683760" Jul 15 22:10:31.846974 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.846935 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="af4119c8-f53e-430e-af20-cf475397e2c7" containerName="test-curl-7869-140337491683760" Jul 15 22:10:31.847013 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.847005 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="af4119c8-f53e-430e-af20-cf475397e2c7" containerName="test-curl-7869-140337491683760" Jul 15 22:10:31.849942 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.849926 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:31.855687 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.855657 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:31.926150 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:31.926117 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56krg\" (UniqueName: \"kubernetes.io/projected/a6995b6a-0010-4d9b-aa78-00734bb0f536-kube-api-access-56krg\") pod \"test-curl-7869-140337491492128\" (UID: \"a6995b6a-0010-4d9b-aa78-00734bb0f536\") " pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:32.027472 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.027429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56krg\" (UniqueName: \"kubernetes.io/projected/a6995b6a-0010-4d9b-aa78-00734bb0f536-kube-api-access-56krg\") pod \"test-curl-7869-140337491492128\" (UID: \"a6995b6a-0010-4d9b-aa78-00734bb0f536\") " pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:32.035549 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.035515 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56krg\" (UniqueName: \"kubernetes.io/projected/a6995b6a-0010-4d9b-aa78-00734bb0f536-kube-api-access-56krg\") pod \"test-curl-7869-140337491492128\" (UID: \"a6995b6a-0010-4d9b-aa78-00734bb0f536\") " pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:32.062451 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.062421 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af4119c8-f53e-430e-af20-cf475397e2c7" path="/var/lib/kubelet/pods/af4119c8-f53e-430e-af20-cf475397e2c7/volumes" Jul 15 22:10:32.160534 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.160423 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:32.297060 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.297030 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:32.299208 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:10:32.299160 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6995b6a_0010_4d9b_aa78_00734bb0f536.slice/crio-b0201229f6d69161371e995a32f43d9ce4913f70644bbde76a4b63e5592c92fa WatchSource:0}: Error finding container b0201229f6d69161371e995a32f43d9ce4913f70644bbde76a4b63e5592c92fa: Status 404 returned error can't find the container with id b0201229f6d69161371e995a32f43d9ce4913f70644bbde76a4b63e5592c92fa Jul 15 22:10:32.512439 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.512397 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491492128" event={"ID":"a6995b6a-0010-4d9b-aa78-00734bb0f536","Type":"ContainerStarted","Data":"b0201229f6d69161371e995a32f43d9ce4913f70644bbde76a4b63e5592c92fa"} Jul 15 22:10:32.513618 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.513597 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491683760" Jul 15 22:10:32.513618 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:32.513607 2572 scope.go:117] "RemoveContainer" containerID="bb15c5b8e999436c5c3105736c41d9d8d63ba689d7a4ff3a9479be40acdb12d6" Jul 15 22:10:33.520502 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:33.520466 2572 generic.go:356] "Generic (PLEG): container finished" podID="a6995b6a-0010-4d9b-aa78-00734bb0f536" containerID="291fe05ca1b19072b5cc9f0d11dd232218db34b95f162aed129b552414b67e8d" exitCode=0 Jul 15 22:10:33.520880 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:33.520508 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491492128" event={"ID":"a6995b6a-0010-4d9b-aa78-00734bb0f536","Type":"ContainerDied","Data":"291fe05ca1b19072b5cc9f0d11dd232218db34b95f162aed129b552414b67e8d"} Jul 15 22:10:34.653937 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.653912 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:34.673195 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.673149 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7869-140337491492128_a6995b6a-0010-4d9b-aa78-00734bb0f536/test-curl-7869-140337491492128/0.log" Jul 15 22:10:34.696871 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.696842 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:34.700738 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.700715 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:34.751341 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.751300 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56krg\" (UniqueName: \"kubernetes.io/projected/a6995b6a-0010-4d9b-aa78-00734bb0f536-kube-api-access-56krg\") pod \"a6995b6a-0010-4d9b-aa78-00734bb0f536\" (UID: \"a6995b6a-0010-4d9b-aa78-00734bb0f536\") " Jul 15 22:10:34.753376 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.753348 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6995b6a-0010-4d9b-aa78-00734bb0f536-kube-api-access-56krg" (OuterVolumeSpecName: "kube-api-access-56krg") pod "a6995b6a-0010-4d9b-aa78-00734bb0f536" (UID: "a6995b6a-0010-4d9b-aa78-00734bb0f536"). InnerVolumeSpecName "kube-api-access-56krg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:34.817597 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.817530 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:34.817884 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.817871 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6995b6a-0010-4d9b-aa78-00734bb0f536" containerName="test-curl-7869-140337491492128" Jul 15 22:10:34.817927 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.817886 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6995b6a-0010-4d9b-aa78-00734bb0f536" containerName="test-curl-7869-140337491492128" Jul 15 22:10:34.817961 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.817944 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6995b6a-0010-4d9b-aa78-00734bb0f536" containerName="test-curl-7869-140337491492128" Jul 15 22:10:34.820757 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.820739 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:34.824504 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.824481 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:34.852002 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.851980 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56krg\" (UniqueName: \"kubernetes.io/projected/a6995b6a-0010-4d9b-aa78-00734bb0f536-kube-api-access-56krg\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:34.953379 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:34.953341 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64vr\" (UniqueName: \"kubernetes.io/projected/ec096444-bd00-4ea4-bd79-96c0efb3d3c2-kube-api-access-z64vr\") pod \"test-curl-7869-140337491492128\" (UID: \"ec096444-bd00-4ea4-bd79-96c0efb3d3c2\") " pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:35.053840 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.053800 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64vr\" (UniqueName: \"kubernetes.io/projected/ec096444-bd00-4ea4-bd79-96c0efb3d3c2-kube-api-access-z64vr\") pod \"test-curl-7869-140337491492128\" (UID: \"ec096444-bd00-4ea4-bd79-96c0efb3d3c2\") " pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:35.061521 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.061491 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64vr\" (UniqueName: \"kubernetes.io/projected/ec096444-bd00-4ea4-bd79-96c0efb3d3c2-kube-api-access-z64vr\") pod \"test-curl-7869-140337491492128\" (UID: \"ec096444-bd00-4ea4-bd79-96c0efb3d3c2\") " pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:35.131771 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.131686 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:35.259202 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.259159 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:35.260732 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:10:35.260702 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec096444_bd00_4ea4_bd79_96c0efb3d3c2.slice/crio-5bd2561192dff4ee4c2d74107fdceaac59ba442a244609575fffc2c63d570e64 WatchSource:0}: Error finding container 5bd2561192dff4ee4c2d74107fdceaac59ba442a244609575fffc2c63d570e64: Status 404 returned error can't find the container with id 5bd2561192dff4ee4c2d74107fdceaac59ba442a244609575fffc2c63d570e64 Jul 15 22:10:35.532146 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.532110 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:35.532146 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.532143 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0201229f6d69161371e995a32f43d9ce4913f70644bbde76a4b63e5592c92fa" Jul 15 22:10:35.533337 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.533310 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491492128" event={"ID":"ec096444-bd00-4ea4-bd79-96c0efb3d3c2","Type":"ContainerStarted","Data":"5bd2561192dff4ee4c2d74107fdceaac59ba442a244609575fffc2c63d570e64"} Jul 15 22:10:35.543466 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:35.543440 2572 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7869-140337491492128" oldPodUID="a6995b6a-0010-4d9b-aa78-00734bb0f536" podUID="ec096444-bd00-4ea4-bd79-96c0efb3d3c2" Jul 15 22:10:36.062240 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:36.062199 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6995b6a-0010-4d9b-aa78-00734bb0f536" path="/var/lib/kubelet/pods/a6995b6a-0010-4d9b-aa78-00734bb0f536/volumes" Jul 15 22:10:36.539639 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:36.539601 2572 generic.go:356] "Generic (PLEG): container finished" podID="ec096444-bd00-4ea4-bd79-96c0efb3d3c2" containerID="9a4fb3e63e8c2b0d4b02b40fe8133537bb49ad43a12ecb982158f40349673d4e" exitCode=0 Jul 15 22:10:36.541694 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:36.539691 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491492128" event={"ID":"ec096444-bd00-4ea4-bd79-96c0efb3d3c2","Type":"ContainerDied","Data":"9a4fb3e63e8c2b0d4b02b40fe8133537bb49ad43a12ecb982158f40349673d4e"} Jul 15 22:10:36.732327 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:36.732293 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:10:37.677933 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.677910 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:37.696484 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.696456 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7869-140337491492128_ec096444-bd00-4ea4-bd79-96c0efb3d3c2/test-curl-7869-140337491492128/0.log" Jul 15 22:10:37.720792 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.720758 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:37.725880 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.725854 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7869-140337491492128"] Jul 15 22:10:37.777825 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.777787 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z64vr\" (UniqueName: \"kubernetes.io/projected/ec096444-bd00-4ea4-bd79-96c0efb3d3c2-kube-api-access-z64vr\") pod \"ec096444-bd00-4ea4-bd79-96c0efb3d3c2\" (UID: \"ec096444-bd00-4ea4-bd79-96c0efb3d3c2\") " Jul 15 22:10:37.779934 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.779909 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec096444-bd00-4ea4-bd79-96c0efb3d3c2-kube-api-access-z64vr" (OuterVolumeSpecName: "kube-api-access-z64vr") pod "ec096444-bd00-4ea4-bd79-96c0efb3d3c2" (UID: "ec096444-bd00-4ea4-bd79-96c0efb3d3c2"). InnerVolumeSpecName "kube-api-access-z64vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:37.837949 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.837874 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:37.838213 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.838200 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec096444-bd00-4ea4-bd79-96c0efb3d3c2" containerName="test-curl-7869-140337491492128" Jul 15 22:10:37.838276 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.838215 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec096444-bd00-4ea4-bd79-96c0efb3d3c2" containerName="test-curl-7869-140337491492128" Jul 15 22:10:37.838276 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.838269 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec096444-bd00-4ea4-bd79-96c0efb3d3c2" containerName="test-curl-7869-140337491492128" Jul 15 22:10:37.840958 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.840942 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:37.847333 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.847307 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:37.878882 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.878853 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64vr\" (UniqueName: \"kubernetes.io/projected/ec096444-bd00-4ea4-bd79-96c0efb3d3c2-kube-api-access-z64vr\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:37.980323 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:37.980279 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8xh9\" (UniqueName: \"kubernetes.io/projected/d76d1d34-3999-4a08-a583-5eda129f2172-kube-api-access-p8xh9\") pod \"test-curl-7869-140337491534160\" (UID: \"d76d1d34-3999-4a08-a583-5eda129f2172\") " pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:38.061898 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.061857 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec096444-bd00-4ea4-bd79-96c0efb3d3c2" path="/var/lib/kubelet/pods/ec096444-bd00-4ea4-bd79-96c0efb3d3c2/volumes" Jul 15 22:10:38.080890 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.080850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8xh9\" (UniqueName: \"kubernetes.io/projected/d76d1d34-3999-4a08-a583-5eda129f2172-kube-api-access-p8xh9\") pod \"test-curl-7869-140337491534160\" (UID: \"d76d1d34-3999-4a08-a583-5eda129f2172\") " pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:38.088505 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.088446 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8xh9\" (UniqueName: \"kubernetes.io/projected/d76d1d34-3999-4a08-a583-5eda129f2172-kube-api-access-p8xh9\") pod \"test-curl-7869-140337491534160\" (UID: \"d76d1d34-3999-4a08-a583-5eda129f2172\") " pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:38.150816 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.150779 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:38.276860 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.276836 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:38.278890 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:10:38.278865 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd76d1d34_3999_4a08_a583_5eda129f2172.slice/crio-6e3365b68443caeab2b35d8929f1b79499fa959b3f7f5a45540a9b4c9a0d98d7 WatchSource:0}: Error finding container 6e3365b68443caeab2b35d8929f1b79499fa959b3f7f5a45540a9b4c9a0d98d7: Status 404 returned error can't find the container with id 6e3365b68443caeab2b35d8929f1b79499fa959b3f7f5a45540a9b4c9a0d98d7 Jul 15 22:10:38.348527 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.348449 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:38.453755 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.453722 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:38.550567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.550538 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491492128" Jul 15 22:10:38.550760 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.550572 2572 scope.go:117] "RemoveContainer" containerID="9a4fb3e63e8c2b0d4b02b40fe8133537bb49ad43a12ecb982158f40349673d4e" Jul 15 22:10:38.551915 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:38.551753 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491534160" event={"ID":"d76d1d34-3999-4a08-a583-5eda129f2172","Type":"ContainerStarted","Data":"6e3365b68443caeab2b35d8929f1b79499fa959b3f7f5a45540a9b4c9a0d98d7"} Jul 15 22:10:39.097495 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.097436 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" podUID="fbc303eb-532b-4219-8b16-79b23ed5a1a0" containerName="main" containerID="cri-o://f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1" gracePeriod=45 Jul 15 22:10:39.353595 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.353532 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:10:39.494068 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494025 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-model-cache\") pod \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " Jul 15 22:10:39.494068 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494061 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-dshm\") pod \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " Jul 15 22:10:39.494333 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494116 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tls-certs\") pod \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " Jul 15 22:10:39.494333 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494189 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmvgc\" (UniqueName: \"kubernetes.io/projected/fbc303eb-532b-4219-8b16-79b23ed5a1a0-kube-api-access-gmvgc\") pod \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " Jul 15 22:10:39.494333 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494241 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tmp-dir\") pod \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " Jul 15 22:10:39.494333 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494272 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-home\") pod \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\" (UID: \"fbc303eb-532b-4219-8b16-79b23ed5a1a0\") " Jul 15 22:10:39.494539 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494390 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "fbc303eb-532b-4219-8b16-79b23ed5a1a0" (UID: "fbc303eb-532b-4219-8b16-79b23ed5a1a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:39.494597 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494558 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:39.494597 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494526 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fbc303eb-532b-4219-8b16-79b23ed5a1a0" (UID: "fbc303eb-532b-4219-8b16-79b23ed5a1a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:39.494675 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.494607 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-home" (OuterVolumeSpecName: "home") pod "fbc303eb-532b-4219-8b16-79b23ed5a1a0" (UID: "fbc303eb-532b-4219-8b16-79b23ed5a1a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:39.496409 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.496388 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc303eb-532b-4219-8b16-79b23ed5a1a0-kube-api-access-gmvgc" (OuterVolumeSpecName: "kube-api-access-gmvgc") pod "fbc303eb-532b-4219-8b16-79b23ed5a1a0" (UID: "fbc303eb-532b-4219-8b16-79b23ed5a1a0"). InnerVolumeSpecName "kube-api-access-gmvgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:39.496484 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.496420 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-dshm" (OuterVolumeSpecName: "dshm") pod "fbc303eb-532b-4219-8b16-79b23ed5a1a0" (UID: "fbc303eb-532b-4219-8b16-79b23ed5a1a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:39.496484 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.496421 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fbc303eb-532b-4219-8b16-79b23ed5a1a0" (UID: "fbc303eb-532b-4219-8b16-79b23ed5a1a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:10:39.559381 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.559345 2572 generic.go:356] "Generic (PLEG): container finished" podID="fbc303eb-532b-4219-8b16-79b23ed5a1a0" containerID="f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1" exitCode=0 Jul 15 22:10:39.559381 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.559383 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" Jul 15 22:10:39.559595 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.559432 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" event={"ID":"fbc303eb-532b-4219-8b16-79b23ed5a1a0","Type":"ContainerDied","Data":"f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1"} Jul 15 22:10:39.559595 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.559475 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84" event={"ID":"fbc303eb-532b-4219-8b16-79b23ed5a1a0","Type":"ContainerDied","Data":"a3d65badfeb982ff195948d343a3ea74b1fcbb7d314794660783594a5b3d8df7"} Jul 15 22:10:39.559595 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.559495 2572 scope.go:117] "RemoveContainer" containerID="f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1" Jul 15 22:10:39.564668 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.564607 2572 generic.go:356] "Generic (PLEG): container finished" podID="d76d1d34-3999-4a08-a583-5eda129f2172" containerID="d9d294f43c18ea8494d64e35d137a706e2fd305a016d48e050d8587a10ca2097" exitCode=0 Jul 15 22:10:39.564743 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.564689 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491534160" event={"ID":"d76d1d34-3999-4a08-a583-5eda129f2172","Type":"ContainerDied","Data":"d9d294f43c18ea8494d64e35d137a706e2fd305a016d48e050d8587a10ca2097"} Jul 15 22:10:39.572230 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.572209 2572 scope.go:117] "RemoveContainer" containerID="f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1" Jul 15 22:10:39.572474 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:10:39.572457 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1\": container with ID starting with f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1 not found: ID does not exist" containerID="f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1" Jul 15 22:10:39.572521 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.572482 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1"} err="failed to get container status \"f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1\": rpc error: code = NotFound desc = could not find container \"f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1\": container with ID starting with f4df3894342a90a1a9cc6a1bc4ba8ef17a3ef8d11be4863c60a14dff73757eb1 not found: ID does not exist" Jul 15 22:10:39.591398 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.591367 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84"] Jul 15 22:10:39.595259 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.595222 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-8172c4/rate-test-model-8172c4-kserve-6f4c46956c-rbt84"] Jul 15 22:10:39.595519 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.595504 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:39.595566 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.595523 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:39.595566 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.595531 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fbc303eb-532b-4219-8b16-79b23ed5a1a0-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:39.595566 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.595539 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fbc303eb-532b-4219-8b16-79b23ed5a1a0-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:39.595566 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.595548 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmvgc\" (UniqueName: \"kubernetes.io/projected/fbc303eb-532b-4219-8b16-79b23ed5a1a0-kube-api-access-gmvgc\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:39.994872 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:39.994831 2572 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" podUID="2a6ec3b3-9dcc-4a86-8448-84163a525b26" containerName="main" containerID="cri-o://ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8" gracePeriod=45 Jul 15 22:10:40.061760 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.061729 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbc303eb-532b-4219-8b16-79b23ed5a1a0" path="/var/lib/kubelet/pods/fbc303eb-532b-4219-8b16-79b23ed5a1a0/volumes" Jul 15 22:10:40.269754 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.269724 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:10:40.401735 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.401700 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tls-certs\") pod \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " Jul 15 22:10:40.401931 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.401762 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-model-cache\") pod \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " Jul 15 22:10:40.401931 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.401785 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-home\") pod \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " Jul 15 22:10:40.401931 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.401811 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-dshm\") pod \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " Jul 15 22:10:40.401931 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.401862 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tmp-dir\") pod \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " Jul 15 22:10:40.401931 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.401905 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpbkc\" (UniqueName: \"kubernetes.io/projected/2a6ec3b3-9dcc-4a86-8448-84163a525b26-kube-api-access-jpbkc\") pod \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\" (UID: \"2a6ec3b3-9dcc-4a86-8448-84163a525b26\") " Jul 15 22:10:40.402219 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.402069 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-home" (OuterVolumeSpecName: "home") pod "2a6ec3b3-9dcc-4a86-8448-84163a525b26" (UID: "2a6ec3b3-9dcc-4a86-8448-84163a525b26"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:40.402219 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.402079 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a6ec3b3-9dcc-4a86-8448-84163a525b26" (UID: "2a6ec3b3-9dcc-4a86-8448-84163a525b26"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:40.402219 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.402186 2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-model-cache\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:40.402219 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.402204 2572 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-home\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:40.402364 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.402274 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a6ec3b3-9dcc-4a86-8448-84163a525b26" (UID: "2a6ec3b3-9dcc-4a86-8448-84163a525b26"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:40.403986 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.403964 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-dshm" (OuterVolumeSpecName: "dshm") pod "2a6ec3b3-9dcc-4a86-8448-84163a525b26" (UID: "2a6ec3b3-9dcc-4a86-8448-84163a525b26"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 22:10:40.403986 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.403970 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6ec3b3-9dcc-4a86-8448-84163a525b26-kube-api-access-jpbkc" (OuterVolumeSpecName: "kube-api-access-jpbkc") pod "2a6ec3b3-9dcc-4a86-8448-84163a525b26" (UID: "2a6ec3b3-9dcc-4a86-8448-84163a525b26"). InnerVolumeSpecName "kube-api-access-jpbkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:40.404149 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.403988 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a6ec3b3-9dcc-4a86-8448-84163a525b26" (UID: "2a6ec3b3-9dcc-4a86-8448-84163a525b26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 22:10:40.502642 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.502560 2572 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tmp-dir\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:40.502642 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.502587 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpbkc\" (UniqueName: \"kubernetes.io/projected/2a6ec3b3-9dcc-4a86-8448-84163a525b26-kube-api-access-jpbkc\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:40.502642 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.502597 2572 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6ec3b3-9dcc-4a86-8448-84163a525b26-tls-certs\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:40.502642 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.502606 2572 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6ec3b3-9dcc-4a86-8448-84163a525b26-dshm\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:40.572081 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.572046 2572 generic.go:356] "Generic (PLEG): container finished" podID="2a6ec3b3-9dcc-4a86-8448-84163a525b26" containerID="ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8" exitCode=0 Jul 15 22:10:40.572248 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.572095 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" Jul 15 22:10:40.572248 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.572133 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" event={"ID":"2a6ec3b3-9dcc-4a86-8448-84163a525b26","Type":"ContainerDied","Data":"ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8"} Jul 15 22:10:40.572248 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.572193 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc" event={"ID":"2a6ec3b3-9dcc-4a86-8448-84163a525b26","Type":"ContainerDied","Data":"99342cd5832c3a1d3269757152fa1b171188b7cb13bfe40f5324035741718fba"} Jul 15 22:10:40.572248 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.572213 2572 scope.go:117] "RemoveContainer" containerID="ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8" Jul 15 22:10:40.581142 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.581125 2572 scope.go:117] "RemoveContainer" containerID="ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8" Jul 15 22:10:40.581409 ip-10-0-135-178 kubenswrapper[2572]: E0715 22:10:40.581380 2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8\": container with ID starting with ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8 not found: ID does not exist" containerID="ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8" Jul 15 22:10:40.581453 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.581408 2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8"} err="failed to get container status \"ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8\": rpc error: code = NotFound desc = could not find container \"ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8\": container with ID starting with ee11919c1a3e0399cc358c8a8f55de1b99b666ec40c35c6f0ad1fed28d618cc8 not found: ID does not exist" Jul 15 22:10:40.592792 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.592762 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc"] Jul 15 22:10:40.596329 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.596297 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8cc1f1/rate-test-model-8cc1f1-kserve-7679878b7f-94lcc"] Jul 15 22:10:40.706427 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.706402 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:40.724618 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.724570 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7869-140337491534160_d76d1d34-3999-4a08-a583-5eda129f2172/test-curl-7869-140337491534160/0.log" Jul 15 22:10:40.746319 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.746286 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:40.748063 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.748041 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:40.805342 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.805254 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8xh9\" (UniqueName: \"kubernetes.io/projected/d76d1d34-3999-4a08-a583-5eda129f2172-kube-api-access-p8xh9\") pod \"d76d1d34-3999-4a08-a583-5eda129f2172\" (UID: \"d76d1d34-3999-4a08-a583-5eda129f2172\") " Jul 15 22:10:40.807335 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.807311 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d76d1d34-3999-4a08-a583-5eda129f2172-kube-api-access-p8xh9" (OuterVolumeSpecName: "kube-api-access-p8xh9") pod "d76d1d34-3999-4a08-a583-5eda129f2172" (UID: "d76d1d34-3999-4a08-a583-5eda129f2172"). InnerVolumeSpecName "kube-api-access-p8xh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:40.870603 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870575 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:40.870875 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870864 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbc303eb-532b-4219-8b16-79b23ed5a1a0" containerName="main" Jul 15 22:10:40.870913 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870877 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbc303eb-532b-4219-8b16-79b23ed5a1a0" containerName="main" Jul 15 22:10:40.870913 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870886 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6ec3b3-9dcc-4a86-8448-84163a525b26" containerName="main" Jul 15 22:10:40.870913 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870892 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6ec3b3-9dcc-4a86-8448-84163a525b26" containerName="main" Jul 15 22:10:40.870913 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870899 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d76d1d34-3999-4a08-a583-5eda129f2172" containerName="test-curl-7869-140337491534160" Jul 15 22:10:40.870913 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870906 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d76d1d34-3999-4a08-a583-5eda129f2172" containerName="test-curl-7869-140337491534160" Jul 15 22:10:40.871085 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870968 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbc303eb-532b-4219-8b16-79b23ed5a1a0" containerName="main" Jul 15 22:10:40.871085 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870976 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d76d1d34-3999-4a08-a583-5eda129f2172" containerName="test-curl-7869-140337491534160" Jul 15 22:10:40.871085 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.870983 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6ec3b3-9dcc-4a86-8448-84163a525b26" containerName="main" Jul 15 22:10:40.873663 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.873645 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:40.880271 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.880244 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:40.906734 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:40.906709 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8xh9\" (UniqueName: \"kubernetes.io/projected/d76d1d34-3999-4a08-a583-5eda129f2172-kube-api-access-p8xh9\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:41.008094 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.008059 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cf76\" (UniqueName: \"kubernetes.io/projected/dc3e2dad-b969-4ee0-a986-21ed39823a1d-kube-api-access-7cf76\") pod \"test-curl-7869-140337491534160\" (UID: \"dc3e2dad-b969-4ee0-a986-21ed39823a1d\") " pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:41.109124 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.109045 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cf76\" (UniqueName: \"kubernetes.io/projected/dc3e2dad-b969-4ee0-a986-21ed39823a1d-kube-api-access-7cf76\") pod \"test-curl-7869-140337491534160\" (UID: \"dc3e2dad-b969-4ee0-a986-21ed39823a1d\") " pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:41.117212 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.117159 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cf76\" (UniqueName: \"kubernetes.io/projected/dc3e2dad-b969-4ee0-a986-21ed39823a1d-kube-api-access-7cf76\") pod \"test-curl-7869-140337491534160\" (UID: \"dc3e2dad-b969-4ee0-a986-21ed39823a1d\") " pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:41.184450 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.184414 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:41.310218 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.310189 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:41.312766 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:10:41.312736 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc3e2dad_b969_4ee0_a986_21ed39823a1d.slice/crio-c062478956969922cad4348afd5196469891fa51b1735e3918e437a5ab4b15e7 WatchSource:0}: Error finding container c062478956969922cad4348afd5196469891fa51b1735e3918e437a5ab4b15e7: Status 404 returned error can't find the container with id c062478956969922cad4348afd5196469891fa51b1735e3918e437a5ab4b15e7 Jul 15 22:10:41.580109 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.580082 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:41.580109 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.580100 2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3365b68443caeab2b35d8929f1b79499fa959b3f7f5a45540a9b4c9a0d98d7" Jul 15 22:10:41.581316 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.581286 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491534160" event={"ID":"dc3e2dad-b969-4ee0-a986-21ed39823a1d","Type":"ContainerStarted","Data":"c062478956969922cad4348afd5196469891fa51b1735e3918e437a5ab4b15e7"} Jul 15 22:10:41.594312 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:41.594286 2572 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7869-140337491534160" oldPodUID="d76d1d34-3999-4a08-a583-5eda129f2172" podUID="dc3e2dad-b969-4ee0-a986-21ed39823a1d" Jul 15 22:10:42.062387 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:42.062352 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6ec3b3-9dcc-4a86-8448-84163a525b26" path="/var/lib/kubelet/pods/2a6ec3b3-9dcc-4a86-8448-84163a525b26/volumes" Jul 15 22:10:42.062681 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:42.062669 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d76d1d34-3999-4a08-a583-5eda129f2172" path="/var/lib/kubelet/pods/d76d1d34-3999-4a08-a583-5eda129f2172/volumes" Jul 15 22:10:42.587520 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:42.587484 2572 generic.go:356] "Generic (PLEG): container finished" podID="dc3e2dad-b969-4ee0-a986-21ed39823a1d" containerID="252f0460d4cde91f0faca2763de6bcca833987e84babb15325db46acb8ca1d61" exitCode=0 Jul 15 22:10:42.587925 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:42.587560 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7869-140337491534160" event={"ID":"dc3e2dad-b969-4ee0-a986-21ed39823a1d","Type":"ContainerDied","Data":"252f0460d4cde91f0faca2763de6bcca833987e84babb15325db46acb8ca1d61"} Jul 15 22:10:43.723555 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.723528 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:43.741390 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.741360 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7869-140337491534160_dc3e2dad-b969-4ee0-a986-21ed39823a1d/test-curl-7869-140337491534160/0.log" Jul 15 22:10:43.764648 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.764620 2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:43.767719 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.767693 2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7869-140337491534160"] Jul 15 22:10:43.831947 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.831913 2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cf76\" (UniqueName: \"kubernetes.io/projected/dc3e2dad-b969-4ee0-a986-21ed39823a1d-kube-api-access-7cf76\") pod \"dc3e2dad-b969-4ee0-a986-21ed39823a1d\" (UID: \"dc3e2dad-b969-4ee0-a986-21ed39823a1d\") " Jul 15 22:10:43.833989 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.833964 2572 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc3e2dad-b969-4ee0-a986-21ed39823a1d-kube-api-access-7cf76" (OuterVolumeSpecName: "kube-api-access-7cf76") pod "dc3e2dad-b969-4ee0-a986-21ed39823a1d" (UID: "dc3e2dad-b969-4ee0-a986-21ed39823a1d"). InnerVolumeSpecName "kube-api-access-7cf76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 22:10:43.933399 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:43.933314 2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cf76\" (UniqueName: \"kubernetes.io/projected/dc3e2dad-b969-4ee0-a986-21ed39823a1d-kube-api-access-7cf76\") on node \"ip-10-0-135-178.ec2.internal\" DevicePath \"\"" Jul 15 22:10:44.062534 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:44.062500 2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc3e2dad-b969-4ee0-a986-21ed39823a1d" path="/var/lib/kubelet/pods/dc3e2dad-b969-4ee0-a986-21ed39823a1d/volumes" Jul 15 22:10:44.598597 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:44.598568 2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7869-140337491534160" Jul 15 22:10:44.598597 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:44.598593 2572 scope.go:117] "RemoveContainer" containerID="252f0460d4cde91f0faca2763de6bcca833987e84babb15325db46acb8ca1d61" Jul 15 22:10:46.831221 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:46.831182 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:48.352228 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:48.352189 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:54.514055 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:54.514010 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:54.614717 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:54.614676 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:54.774966 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:54.774880 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:10:58.325885 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:58.325853 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:10:59.050651 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:10:59.050612 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:04.792978 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:04.792937 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:04.855450 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:04.855411 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:09.737114 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:09.737074 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:09.979465 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:09.979425 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:10.617998 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:10.617965 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:10.719054 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:10.719011 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:20.686058 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:20.686026 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:24.126016 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:24.125980 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-6jhcv"] Jul 15 22:11:25.191921 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:25.191881 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:31.049358 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:31.049324 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:31.106180 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:31.106140 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:35.859897 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:35.859858 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:36.091063 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:36.091007 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:36.524150 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:36.524110 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b68dcc544-lfb7k_6fb7d29a-977c-4c7b-b48a-1d99d3c99c58/authorino/0.log" Jul 15 22:11:42.386918 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:42.386893 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p7dsb_b271d7a1-43b6-4621-871c-b3ecb87b9817/server/0.log" Jul 15 22:11:42.514397 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:42.514364 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wccdv_8041204f-ce83-4e43-8321-1ce6fffc7ef1/manager/0.log" Jul 15 22:11:42.769090 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:42.769065 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fdfd95c4-9q8fq_2b4d5286-535c-4397-9a32-ef2cc880529a/copy-manifests/0.log" Jul 15 22:11:42.775487 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:42.775463 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fdfd95c4-9q8fq_2b4d5286-535c-4397-9a32-ef2cc880529a/manager/0.log" Jul 15 22:11:45.572014 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.571981 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q_47b2923f-e3ad-408a-9e45-85512fb7f5a0/util/0.log" Jul 15 22:11:45.578357 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.578334 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q_47b2923f-e3ad-408a-9e45-85512fb7f5a0/pull/0.log" Jul 15 22:11:45.584407 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.584386 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q_47b2923f-e3ad-408a-9e45-85512fb7f5a0/extract/0.log" Jul 15 22:11:45.695042 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.695014 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs_bdbac99f-8970-48b4-90d0-d1e47726064c/util/0.log" Jul 15 22:11:45.700825 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.700800 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs_bdbac99f-8970-48b4-90d0-d1e47726064c/pull/0.log" Jul 15 22:11:45.708476 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.708449 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs_bdbac99f-8970-48b4-90d0-d1e47726064c/extract/0.log" Jul 15 22:11:45.812278 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.812247 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt_3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036/util/0.log" Jul 15 22:11:45.817925 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.817906 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt_3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036/pull/0.log" Jul 15 22:11:45.823016 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.822948 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt_3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036/extract/0.log" Jul 15 22:11:45.933119 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.933084 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d_b15bb427-5c77-4b3e-9491-5a2c47c162e7/pull/0.log" Jul 15 22:11:45.940736 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.940712 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d_b15bb427-5c77-4b3e-9491-5a2c47c162e7/extract/0.log" Jul 15 22:11:45.946545 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:45.946526 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d_b15bb427-5c77-4b3e-9491-5a2c47c162e7/util/0.log" Jul 15 22:11:46.063062 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:46.063030 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b68dcc544-lfb7k_6fb7d29a-977c-4c7b-b48a-1d99d3c99c58/authorino/0.log" Jul 15 22:11:46.407093 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:46.407068 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-99drq_c8680ddf-2d96-43b5-965a-b308bb1335f9/kuadrant-console-plugin/0.log" Jul 15 22:11:46.790827 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:46.790776 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6jhcv_f33899f1-4f3f-4b94-b55c-d98fe1bd9554/limitador/0.log" Jul 15 22:11:47.647487 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:47.647454 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bff-fsq5n_9ae24758-303f-40fd-a1cc-9357617871e9/kube-auth-proxy/0.log" Jul 15 22:11:48.933375 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:48.933340 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-srrb5_77493385-61ce-456a-8cd5-9d5bd3239489/main/0.log" Jul 15 22:11:54.928281 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.928242 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rh4xt/must-gather-hbn2v"] Jul 15 22:11:54.928771 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.928561 2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc3e2dad-b969-4ee0-a986-21ed39823a1d" containerName="test-curl-7869-140337491534160" Jul 15 22:11:54.928771 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.928572 2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc3e2dad-b969-4ee0-a986-21ed39823a1d" containerName="test-curl-7869-140337491534160" Jul 15 22:11:54.928771 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.928639 2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc3e2dad-b969-4ee0-a986-21ed39823a1d" containerName="test-curl-7869-140337491534160" Jul 15 22:11:54.931676 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.931656 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:54.934054 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.934033 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rh4xt\"/\"kube-root-ca.crt\"" Jul 15 22:11:54.934356 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.934329 2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rh4xt\"/\"openshift-service-ca.crt\"" Jul 15 22:11:54.943384 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.943348 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rh4xt/must-gather-hbn2v"] Jul 15 22:11:54.950635 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:54.950598 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:55.025044 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.025001 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9fvp\" (UniqueName: \"kubernetes.io/projected/5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a-kube-api-access-h9fvp\") pod \"must-gather-hbn2v\" (UID: \"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a\") " pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.025243 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.025071 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a-must-gather-output\") pod \"must-gather-hbn2v\" (UID: \"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a\") " pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.045926 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.045889 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" limit=1024 Jul 15 22:11:55.125895 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.125861 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9fvp\" (UniqueName: \"kubernetes.io/projected/5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a-kube-api-access-h9fvp\") pod \"must-gather-hbn2v\" (UID: \"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a\") " pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.126064 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.125915 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a-must-gather-output\") pod \"must-gather-hbn2v\" (UID: \"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a\") " pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.126276 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.126260 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a-must-gather-output\") pod \"must-gather-hbn2v\" (UID: \"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a\") " pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.133506 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.133483 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9fvp\" (UniqueName: \"kubernetes.io/projected/5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a-kube-api-access-h9fvp\") pod \"must-gather-hbn2v\" (UID: \"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a\") " pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.242871 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.242834 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rh4xt/must-gather-hbn2v" Jul 15 22:11:55.376750 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.376716 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rh4xt/must-gather-hbn2v"] Jul 15 22:11:55.380112 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:11:55.380087 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ff9e4d4_da2c_4eff_ad71_4f9b8935ea6a.slice/crio-3be2115cd9db9bec07e2e30507dfdee389fc9b3b8143f3ad441aed5f540b67f5 WatchSource:0}: Error finding container 3be2115cd9db9bec07e2e30507dfdee389fc9b3b8143f3ad441aed5f540b67f5: Status 404 returned error can't find the container with id 3be2115cd9db9bec07e2e30507dfdee389fc9b3b8143f3ad441aed5f540b67f5 Jul 15 22:11:55.950547 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:55.950474 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rh4xt/must-gather-hbn2v" event={"ID":"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a","Type":"ContainerStarted","Data":"3be2115cd9db9bec07e2e30507dfdee389fc9b3b8143f3ad441aed5f540b67f5"} Jul 15 22:11:56.959770 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:56.959731 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rh4xt/must-gather-hbn2v" event={"ID":"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a","Type":"ContainerStarted","Data":"fd7ffd1ad6077338691db43d934b716df3261f958ec20a129288a8b926a32bee"} Jul 15 22:11:56.960224 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:56.959792 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rh4xt/must-gather-hbn2v" event={"ID":"5ff9e4d4-da2c-4eff-ad71-4f9b8935ea6a","Type":"ContainerStarted","Data":"90706b3aada8c78c857d9de5b26b6d7d1453ac1cac155cac56b2b0c9ce8b8133"} Jul 15 22:11:56.975825 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:56.975762 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rh4xt/must-gather-hbn2v" podStartSLOduration=2.148916613 podStartE2EDuration="2.975744954s" totalImagesPullingTime="826.828341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 22:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:11:56.972880981 +0000 UTC m=+2513.553478013" watchObservedRunningTime="2026-07-15 22:11:56.975744954 +0000 UTC m=+2513.556341983" Jul 15 22:11:57.775086 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:57.775050 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tg8tn_b1d9bff4-d3f4-47ea-9951-48073e0519a9/global-pull-secret-syncer/0.log" Jul 15 22:11:57.826949 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:57.826902 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j6tlt_e7a08fd4-44f6-4065-b984-e0a1a5554cdd/konnectivity-agent/0.log" Jul 15 22:11:57.963353 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:11:57.963300 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-178.ec2.internal_102fbd48ff7ff0cfcd397c73fd966f70/haproxy/0.log" Jul 15 22:12:02.787333 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.787297 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q_47b2923f-e3ad-408a-9e45-85512fb7f5a0/extract/0.log" Jul 15 22:12:02.807763 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.807735 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q_47b2923f-e3ad-408a-9e45-85512fb7f5a0/util/0.log" Jul 15 22:12:02.828513 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.828472 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759ggv7q_47b2923f-e3ad-408a-9e45-85512fb7f5a0/pull/0.log" Jul 15 22:12:02.854525 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.854439 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs_bdbac99f-8970-48b4-90d0-d1e47726064c/extract/0.log" Jul 15 22:12:02.873780 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.873696 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs_bdbac99f-8970-48b4-90d0-d1e47726064c/util/0.log" Jul 15 22:12:02.899537 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.899506 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05xcfs_bdbac99f-8970-48b4-90d0-d1e47726064c/pull/0.log" Jul 15 22:12:02.924746 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.924722 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt_3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036/extract/0.log" Jul 15 22:12:02.943677 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.943647 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt_3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036/util/0.log" Jul 15 22:12:02.963789 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:02.963757 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73zv9tt_3f84e7a5-99db-4d0d-a6bc-eb7cf18e2036/pull/0.log" Jul 15 22:12:03.003291 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:03.003260 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d_b15bb427-5c77-4b3e-9491-5a2c47c162e7/extract/0.log" Jul 15 22:12:03.029573 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:03.029548 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d_b15bb427-5c77-4b3e-9491-5a2c47c162e7/util/0.log" Jul 15 22:12:03.048417 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:03.048341 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1q5j9d_b15bb427-5c77-4b3e-9491-5a2c47c162e7/pull/0.log" Jul 15 22:12:03.525701 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:03.525518 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5b68dcc544-lfb7k_6fb7d29a-977c-4c7b-b48a-1d99d3c99c58/authorino/0.log" Jul 15 22:12:03.598016 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:03.597979 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-99drq_c8680ddf-2d96-43b5-965a-b308bb1335f9/kuadrant-console-plugin/0.log" Jul 15 22:12:03.882417 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:03.882345 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-6jhcv_f33899f1-4f3f-4b94-b55c-d98fe1bd9554/limitador/0.log" Jul 15 22:12:05.562604 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.562576 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/alertmanager/0.log" Jul 15 22:12:05.582305 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.582277 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/config-reloader/0.log" Jul 15 22:12:05.603848 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.603750 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/kube-rbac-proxy-web/0.log" Jul 15 22:12:05.623688 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.623659 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/kube-rbac-proxy/0.log" Jul 15 22:12:05.646027 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.645999 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/kube-rbac-proxy-metric/0.log" Jul 15 22:12:05.663133 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.663009 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/prom-label-proxy/0.log" Jul 15 22:12:05.686384 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.686358 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_02bfcb67-f844-449d-8682-9e2152580cbf/init-config-reloader/0.log" Jul 15 22:12:05.956825 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.956795 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdkxd_e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a/node-exporter/0.log" Jul 15 22:12:05.973948 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.973918 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdkxd_e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a/kube-rbac-proxy/0.log" Jul 15 22:12:05.992512 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:05.992484 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qdkxd_e0fc3ebe-e8a0-4ef5-ade7-35d187cdd29a/init-textfile/0.log" Jul 15 22:12:07.833567 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.833526 2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849"] Jul 15 22:12:07.837839 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.837812 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:07.840823 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.840801 2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rh4xt\"/\"default-dockercfg-9jmjh\"" Jul 15 22:12:07.848876 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.848849 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849"] Jul 15 22:12:07.949755 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.949719 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-proc\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:07.950043 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.950024 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-sys\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:07.950256 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.950236 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-podres\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:07.950439 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.950422 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-lib-modules\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:07.950574 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:07.950560 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mnsx\" (UniqueName: \"kubernetes.io/projected/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-kube-api-access-7mnsx\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052580 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052548 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-podres\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052750 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052601 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-lib-modules\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052750 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mnsx\" (UniqueName: \"kubernetes.io/projected/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-kube-api-access-7mnsx\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052750 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052666 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-proc\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052750 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052691 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-sys\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052960 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052754 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-podres\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052960 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052777 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-sys\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052960 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052755 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-lib-modules\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.052960 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.052823 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-proc\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.064404 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.064379 2572 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mnsx\" (UniqueName: \"kubernetes.io/projected/4e002f64-5be4-4e28-90be-0a7f83bdcbd9-kube-api-access-7mnsx\") pod \"perf-node-gather-daemonset-fc849\" (UID: \"4e002f64-5be4-4e28-90be-0a7f83bdcbd9\") " pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.154118 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.154025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:08.335194 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:08.334762 2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849"] Jul 15 22:12:08.338641 ip-10-0-135-178 kubenswrapper[2572]: W0715 22:12:08.337691 2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4e002f64_5be4_4e28_90be_0a7f83bdcbd9.slice/crio-83ed1aceea2d1e226ffec17156831efa6e5d36c444c7f55633ae87fae960ef8a WatchSource:0}: Error finding container 83ed1aceea2d1e226ffec17156831efa6e5d36c444c7f55633ae87fae960ef8a: Status 404 returned error can't find the container with id 83ed1aceea2d1e226ffec17156831efa6e5d36c444c7f55633ae87fae960ef8a Jul 15 22:12:09.048662 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:09.048622 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" event={"ID":"4e002f64-5be4-4e28-90be-0a7f83bdcbd9","Type":"ContainerStarted","Data":"f0077f9bc7dab29fd9c640586cf0fea5310d0fb564b95ece480922da2592e35e"} Jul 15 22:12:09.049073 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:09.048692 2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" event={"ID":"4e002f64-5be4-4e28-90be-0a7f83bdcbd9","Type":"ContainerStarted","Data":"83ed1aceea2d1e226ffec17156831efa6e5d36c444c7f55633ae87fae960ef8a"} Jul 15 22:12:09.049073 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:09.049025 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:09.062982 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:09.062926 2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" podStartSLOduration=2.062902955 podStartE2EDuration="2.062902955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 22:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 22:12:09.062139222 +0000 UTC m=+2525.642736314" watchObservedRunningTime="2026-07-15 22:12:09.062902955 +0000 UTC m=+2525.643499985" Jul 15 22:12:10.449855 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:10.449825 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wv84r_bc69e67e-2596-4e81-b465-68ea1ef9fa78/dns/0.log" Jul 15 22:12:10.469468 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:10.469442 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wv84r_bc69e67e-2596-4e81-b465-68ea1ef9fa78/kube-rbac-proxy/0.log" Jul 15 22:12:10.509300 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:10.509272 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-p45dp_7639f6d8-7349-44cd-8ce9-25027172896c/dns-node-resolver/0.log" Jul 15 22:12:11.076594 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:11.076559 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bssvn_ed84b377-e8fa-4089-8a99-79436b6fa9b4/node-ca/0.log" Jul 15 22:12:12.189939 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:12.189911 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bff-fsq5n_9ae24758-303f-40fd-a1cc-9357617871e9/kube-auth-proxy/0.log" Jul 15 22:12:13.028495 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:13.028459 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4rp7g_4fc95afb-8b40-4b9e-8fef-74cc549460ea/serve-healthcheck-canary/0.log" Jul 15 22:12:13.641692 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:13.641653 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-slzq9_80c38c57-d37a-474d-911a-af4fb6182de6/kube-rbac-proxy/0.log" Jul 15 22:12:13.659648 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:13.659615 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-slzq9_80c38c57-d37a-474d-911a-af4fb6182de6/exporter/0.log" Jul 15 22:12:13.679097 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:13.679071 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-slzq9_80c38c57-d37a-474d-911a-af4fb6182de6/extractor/0.log" Jul 15 22:12:15.069859 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:15.069816 2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rh4xt/perf-node-gather-daemonset-fc849" Jul 15 22:12:16.077609 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:16.077575 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p7dsb_b271d7a1-43b6-4621-871c-b3ecb87b9817/server/0.log" Jul 15 22:12:16.181743 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:16.181706 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wccdv_8041204f-ce83-4e43-8321-1ce6fffc7ef1/manager/0.log" Jul 15 22:12:16.325465 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:16.325431 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fdfd95c4-9q8fq_2b4d5286-535c-4397-9a32-ef2cc880529a/manager/0.log" Jul 15 22:12:16.344860 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:16.344774 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56fdfd95c4-9q8fq_2b4d5286-535c-4397-9a32-ef2cc880529a/copy-manifests/0.log" Jul 15 22:12:17.830804 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:17.830765 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6577974b46-4zvd4_55d1bf71-b2a7-4f11-8fcf-fa83e46b13c2/manager/0.log" Jul 15 22:12:17.878316 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:17.878289 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-l92b8_2247ae8a-5566-4f43-a2ca-a3a32e7596ad/openshift-lws-operator/0.log" Jul 15 22:12:18.684124 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:18.684096 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bff-fsq5n_9ae24758-303f-40fd-a1cc-9357617871e9/kube-auth-proxy/0.log" Jul 15 22:12:20.237380 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:20.237349 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9bff-fsq5n_9ae24758-303f-40fd-a1cc-9357617871e9/kube-auth-proxy/0.log" Jul 15 22:12:23.952419 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:23.952386 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-89q4j_3b27b4e0-f870-47ec-a705-c370b082ce9a/kube-multus/0.log" Jul 15 22:12:24.323617 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.323545 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/kube-multus-additional-cni-plugins/0.log" Jul 15 22:12:24.341596 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.341571 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/egress-router-binary-copy/0.log" Jul 15 22:12:24.360344 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.360307 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/cni-plugins/0.log" Jul 15 22:12:24.377936 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.377906 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/bond-cni-plugin/0.log" Jul 15 22:12:24.395328 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.395303 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/routeoverride-cni/0.log" Jul 15 22:12:24.414241 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.414213 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/whereabouts-cni-bincopy/0.log" Jul 15 22:12:24.431209 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.431185 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-tfzw6_bae5c93e-952f-4779-9030-8d96e59ced89/whereabouts-cni/0.log" Jul 15 22:12:24.572891 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.572863 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7qtb7_16cfb664-2ac4-4852-8f0b-85b1422267e6/network-metrics-daemon/0.log" Jul 15 22:12:24.589678 ip-10-0-135-178 kubenswrapper[2572]: I0715 22:12:24.589604 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7qtb7_16cfb664-2ac4-4852-8f0b-85b1422267e6/kube-rbac-proxy/0.log"