Jul 06 22:23:58.147725 ip-10-0-135-55 systemd[1]: Starting Kubernetes Kubelet... Jul 06 22:23:58.548853 ip-10-0-135-55 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:23:58.548853 ip-10-0-135-55 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 22:23:58.548853 ip-10-0-135-55 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:23:58.548853 ip-10-0-135-55 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 22:23:58.548853 ip-10-0-135-55 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 22:23:58.548853 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.544134 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550771 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550797 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550803 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550808 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550813 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550818 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550822 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550827 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550831 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550836 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550840 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550845 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550849 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550853 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550857 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550861 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550868 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550874 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550878 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:23:58.560560 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550882 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550888 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550895 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550899 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550904 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550908 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550915 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550919 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550923 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550928 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550932 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550935 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550940 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550944 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550947 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550951 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550958 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550964 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550968 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:23:58.567328 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550972 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550976 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550981 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550985 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550989 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550993 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.550997 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551003 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551007 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551011 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551015 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551019 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551023 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551028 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551032 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551036 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551039 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551043 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551047 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551051 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:23:58.570730 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551055 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551058 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551062 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551066 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551072 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551076 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551080 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551083 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551088 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551091 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551095 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551099 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551103 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551107 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551112 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551116 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551120 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551124 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551129 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551133 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:23:58.579138 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551137 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551141 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551145 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551149 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551152 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551157 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551161 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551165 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551169 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551172 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551176 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551180 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551184 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551188 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551191 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551196 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551199 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551203 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551207 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:23:58.582823 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551211 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.551215 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553139 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553151 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553156 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553162 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553167 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553175 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553181 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553186 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553190 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553195 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553199 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553203 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553207 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553211 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553215 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553219 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553223 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:23:58.592516 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553228 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553233 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553237 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553240 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553244 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553248 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553252 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553256 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553261 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553265 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553268 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553272 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553276 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553281 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553285 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553289 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553293 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553297 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553302 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553306 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:23:58.595840 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553310 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553314 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553318 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553322 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553326 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553330 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553334 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553338 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553341 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553348 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553352 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553356 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553360 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553365 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553369 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553373 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553378 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553382 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553386 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553390 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:23:58.605615 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553396 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553402 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553406 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553412 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553416 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553476 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553482 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553487 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553490 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553494 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553498 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553502 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553506 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553510 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553514 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553518 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553521 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553525 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553549 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553556 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:23:58.609602 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553561 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553568 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553572 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553576 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553580 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553584 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553589 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553593 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553597 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553601 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553605 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553609 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553613 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553617 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553621 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553624 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553628 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553632 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553636 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553640 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:23:58.624612 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553644 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.553648 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554262 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554270 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554275 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554279 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554285 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554289 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554295 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554301 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554305 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554310 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554315 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554321 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554325 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554329 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554333 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554336 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554342 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:23:58.625871 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554348 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:23:58.625577 ip-10-0-135-55 systemd[1]: Started Kubernetes Kubelet. Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554353 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554357 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554361 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554365 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554369 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554373 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554377 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554381 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554385 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554389 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554393 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554396 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554400 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554404 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554408 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554412 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554416 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554420 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554440 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:23:58.650150 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554444 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554447 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554451 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554456 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554459 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554463 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554469 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554473 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554477 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554481 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554484 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554488 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554494 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554498 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554502 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554508 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554511 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554515 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554519 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554523 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:23:58.654569 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554527 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554531 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554535 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554539 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554542 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554546 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554550 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554555 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554561 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554565 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554569 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554573 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554576 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554580 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554584 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554587 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554591 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554595 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554600 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554606 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:23:58.817200 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554610 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554614 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554618 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554622 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554626 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554630 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554634 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554641 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554645 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554649 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554654 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554657 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554661 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554665 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554668 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554672 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554676 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554680 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554684 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:23:58.835056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554687 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554690 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.554694 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554782 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554792 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554801 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554808 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554814 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554820 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554826 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554832 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554839 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554843 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554849 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554854 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554858 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554863 2564 flags.go:64] FLAG: --cgroup-root="" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554867 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554871 2564 flags.go:64] FLAG: --client-ca-file="" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554876 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554883 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554889 2564 flags.go:64] FLAG: --cluster-domain="" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554893 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 22:23:58.842249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554898 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554903 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554907 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554913 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554917 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554923 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554927 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554932 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554936 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554940 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554945 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554949 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554955 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554960 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554964 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554969 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554973 2564 flags.go:64] FLAG: --enable-server="true" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554977 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554984 2564 flags.go:64] FLAG: --event-burst="100" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554989 2564 flags.go:64] FLAG: --event-qps="50" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554995 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.554999 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555003 2564 flags.go:64] FLAG: --eviction-hard="" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555010 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 22:23:58.845083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555015 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555019 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555024 2564 flags.go:64] FLAG: --eviction-soft="" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555029 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555033 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555040 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555045 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555050 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555054 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555058 2564 flags.go:64] FLAG: --feature-gates="" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555064 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555068 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555073 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555078 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555082 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555086 2564 flags.go:64] FLAG: --help="false" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555091 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-135-55.ec2.internal" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555095 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555100 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555105 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555110 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555115 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555119 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 22:23:58.849172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555124 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555128 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555132 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555137 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555142 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555147 2564 flags.go:64] FLAG: --kube-reserved="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555152 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555156 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555160 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555165 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555169 2564 flags.go:64] FLAG: --lock-file="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555174 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555178 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555183 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555192 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555198 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555202 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555206 2564 flags.go:64] FLAG: --logging-format="text" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555210 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555216 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555220 2564 flags.go:64] FLAG: --manifest-url="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555225 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555231 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555235 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555242 2564 flags.go:64] FLAG: --max-pods="110" Jul 06 22:23:58.849994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555246 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555251 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555255 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555259 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555264 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555268 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555273 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555283 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555288 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555292 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555297 2564 flags.go:64] FLAG: --pod-cidr="" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555301 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555310 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555314 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555319 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555323 2564 flags.go:64] FLAG: --port="10250" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555328 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555332 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06fc70b67bc973280" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555337 2564 flags.go:64] FLAG: --qos-reserved="" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555342 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555346 2564 flags.go:64] FLAG: --register-node="true" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555351 2564 flags.go:64] FLAG: --register-with-taints="" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555358 2564 flags.go:64] FLAG: --registry-burst="10" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555363 2564 flags.go:64] FLAG: --registry-qps="5" Jul 06 22:23:58.851954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555367 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555372 2564 flags.go:64] FLAG: --reserved-memory="" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555377 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555400 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555406 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555411 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555415 2564 flags.go:64] FLAG: --runonce="false" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555419 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555446 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555451 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555456 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555460 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555465 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555470 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555475 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555479 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555483 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555487 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555491 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555496 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555502 2564 flags.go:64] FLAG: --system-cgroups="" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555507 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555515 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555520 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555525 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 22:23:58.852744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555531 2564 flags.go:64] FLAG: --tls-min-version="" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555535 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555540 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555544 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555549 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555556 2564 flags.go:64] FLAG: --v="2" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555562 2564 flags.go:64] FLAG: --version="false" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555571 2564 flags.go:64] FLAG: --vmodule="" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555576 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.555582 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555781 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555787 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555792 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555796 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555800 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555804 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555808 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555812 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555817 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555821 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555825 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555829 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:23:58.853675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555833 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555837 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555840 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555844 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555848 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555854 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555858 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555862 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555866 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555869 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555873 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555878 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555882 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555886 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555889 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555895 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555901 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555906 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555912 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:23:58.854346 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555917 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555921 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555925 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555929 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555933 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555938 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555942 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555945 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555950 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555954 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555957 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555961 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555965 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555969 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555973 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555976 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555980 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555984 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555990 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555994 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:23:58.855069 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.555998 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556002 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556006 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556009 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556013 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556019 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556023 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556027 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556033 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556037 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556041 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556045 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556050 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556054 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556058 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556062 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556065 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556069 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556073 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556077 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:23:58.855783 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556081 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556085 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556089 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556093 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556097 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556100 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556105 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556108 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556112 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556116 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556121 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556126 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556130 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556134 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556139 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556144 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556150 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556155 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556159 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:23:58.857075 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556163 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556169 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556173 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556178 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556181 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556185 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556190 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556194 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.556198 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.556851 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569070 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569086 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569104 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569110 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569168 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:23:58.857908 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569173 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569178 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569182 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569185 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569189 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569192 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569195 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569198 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569201 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569204 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569207 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569209 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569212 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569215 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569217 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569220 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569222 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569225 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569228 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569231 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:23:59.063089 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569233 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569236 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569239 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569242 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569245 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569248 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569250 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569253 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569255 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569258 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569261 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569263 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569265 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569268 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569270 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569273 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569275 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569278 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569280 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569283 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:23:59.063905 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569285 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569288 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569290 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569294 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569297 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569300 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569303 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569305 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569308 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569310 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569315 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569318 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569323 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569326 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569329 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569332 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569335 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569338 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569340 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:23:59.064718 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569343 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569346 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569348 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569351 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569354 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569357 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569359 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569362 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569364 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569367 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569369 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569372 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569374 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569377 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569379 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569382 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569385 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569387 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569389 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569392 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:23:59.067056 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569394 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569397 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569399 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569403 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569406 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569409 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569411 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569414 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569416 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569419 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569436 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569439 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569442 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569444 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569447 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569450 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569452 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569455 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:23:59.067686 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569457 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569463 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569573 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569577 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569580 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569582 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569585 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569588 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569591 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569595 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569598 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569601 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569604 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569607 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569609 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 22:23:59.068207 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569612 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569615 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569619 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569621 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569624 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569627 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569630 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569632 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569635 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569637 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569640 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569643 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569645 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569648 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569651 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569653 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569655 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569658 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569660 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569663 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 22:23:59.068789 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569665 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569668 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569671 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569673 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569675 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569679 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569683 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569685 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569688 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569691 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569693 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569696 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569699 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569704 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569709 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569711 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569714 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569716 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569719 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 22:23:59.069355 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569721 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569724 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569726 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569729 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569732 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569735 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569738 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569740 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569743 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569745 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569748 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569750 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569753 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569756 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569758 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569761 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569763 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569766 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569768 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569770 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 22:23:59.070082 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569773 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569775 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569778 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569780 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569783 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569785 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569789 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569793 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569795 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569798 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569801 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569803 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569805 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569808 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569811 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569813 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569816 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569819 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569822 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569824 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 22:23:59.070677 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569828 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569831 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569834 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569837 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569839 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569842 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:58.569844 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569849 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.569942 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.571929 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.573143 2564 server.go:1019] "Starting client certificate rotation" Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.575559 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.575787 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:23:59.245649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.575827 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 22:23:59.246070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.578435 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 22:23:59.246070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.591204 2564 log.go:25] "Validated CRI v1 runtime API" Jul 06 22:23:59.246070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.595340 2564 log.go:25] "Validated CRI v1 image API" Jul 06 22:23:59.246070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.596912 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 22:23:59.246070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.599195 2564 fs.go:135] Filesystem UUIDs: map[5316dfd8-1974-4ed8-9812-b032968a0d7d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 e4711dcd-ae3e-413b-80ca-8c823325e663:/dev/nvme0n1p4] Jul 06 22:23:59.246070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.599210 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.604087 2564 manager.go:216] Machine: {Timestamp:2026-07-06 22:23:58.602897335 +0000 UTC m=+0.339299313 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104352 MemoryCapacity:33164505088 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29507a620054c48fd93a3334c9bd49 SystemUUID:ec29507a-6200-54c4-8fd9-3a3334c9bd49 BootID:3ddbde27-e7ac-49a3-905b-46cec0c1312e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582254592 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bd:1f:07:49:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bd:1f:07:49:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:6c:0b:1d:c8:5d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164505088 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.604212 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.604315 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.606174 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.607200 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.607227 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-55.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.608126 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.608138 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.608145 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.608748 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.609891 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.612816 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.612934 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.612963 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.612980 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.612995 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 22:23:59.453029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.613004 2564 kubelet.go:398] "Adding apiserver pod source" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.613017 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.613346 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.613360 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.615555 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.617884 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.617926 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.618989 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619016 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619028 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619035 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619049 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619056 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619493 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619500 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619506 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619523 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619540 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.619548 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.620340 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.620353 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.624775 2564 server.go:1268] "Started kubelet" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.624852 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.625161 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.625213 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.630312 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.630812 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l58d8" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.630971 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-55.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.631346 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-55.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.631643 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.632096 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.632357 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.633738 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l58d8" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.639697 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.640207 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.640314 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.640558 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.640573 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.640671 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.640747 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 06 22:23:59.453661 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.640756 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.639632 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-55.ec2.internal.18bfd39aa5f1f2ed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-55.ec2.internal,UID:ip-10-0-135-55.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-55.ec2.internal,},FirstTimestamp:2026-07-06 22:23:58.624527085 +0000 UTC m=+0.360929062,LastTimestamp:2026-07-06 22:23:58.624527085 +0000 UTC m=+0.360929062,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-55.ec2.internal,}" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.648679 2564 factory.go:55] Registering systemd factory Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.648695 2564 factory.go:223] Registration of the systemd container factory successfully Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.648912 2564 factory.go:153] Registering CRI-O factory Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.648925 2564 factory.go:223] Registration of the crio container factory successfully Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.648983 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.649010 2564 factory.go:103] Registering Raw factory Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.649022 2564 manager.go:1196] Started watching for new ooms in manager Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.649449 2564 manager.go:349] Starting recovery of all containers Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.654206 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.656212 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.658716 2564 manager.go:354] Recovery completed Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.662923 2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-55.ec2.internal\" not found" node="ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.667652 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670024 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670052 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670067 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670690 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670702 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670700 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.670751 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.671581 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.671606 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.671620 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.740576 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.803461 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.803510 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.803535 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.803577 2564 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.806215 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.815305 2564 policy_none.go:49] "None policy: Start" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.815320 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.815330 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.816643 2564 policy_none.go:47] "Start" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.851835 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858273 2564 manager.go:337] "Starting Device Plugin manager" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.858304 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858315 2564 server.go:87] "Starting device plugin registration server" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858630 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858643 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858736 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858840 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.858850 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 22:23:59.455275 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.859453 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.859491 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.904339 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal"] Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.904408 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.905244 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.905268 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.905278 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.906720 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.906841 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.906869 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.910166 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.910193 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.910167 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.910211 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.910238 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.910253 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.911400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.911438 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.912153 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.912180 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.912189 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.947198 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-55.ec2.internal\" not found" node="ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.951519 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-55.ec2.internal\" not found" node="ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.951972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6383cbabd87298deaa3c473fa7d5636c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal\" (UID: \"6383cbabd87298deaa3c473fa7d5636c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.951996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6383cbabd87298deaa3c473fa7d5636c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal\" (UID: \"6383cbabd87298deaa3c473fa7d5636c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.952016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86df2e047708b9ac3980718e830e2fcf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-55.ec2.internal\" (UID: \"86df2e047708b9ac3980718e830e2fcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.959410 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.960232 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientMemory" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.960272 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.960285 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeHasSufficientPID" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.960312 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:58.965692 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.965708 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-55.ec2.internal\": node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.456515 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:58.979128 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.052915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6383cbabd87298deaa3c473fa7d5636c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal\" (UID: \"6383cbabd87298deaa3c473fa7d5636c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.052946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6383cbabd87298deaa3c473fa7d5636c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal\" (UID: \"6383cbabd87298deaa3c473fa7d5636c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.052967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86df2e047708b9ac3980718e830e2fcf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-55.ec2.internal\" (UID: \"86df2e047708b9ac3980718e830e2fcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.053012 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6383cbabd87298deaa3c473fa7d5636c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal\" (UID: \"6383cbabd87298deaa3c473fa7d5636c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.053012 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6383cbabd87298deaa3c473fa7d5636c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal\" (UID: \"6383cbabd87298deaa3c473fa7d5636c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.053005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/86df2e047708b9ac3980718e830e2fcf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-55.ec2.internal\" (UID: \"86df2e047708b9ac3980718e830e2fcf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.080124 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.180740 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.250032 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.254491 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.281524 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.457609 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.382038 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.482534 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.482503 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-55.ec2.internal\" not found" Jul 06 22:23:59.489909 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.489885 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:23:59.540817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.540789 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.559259 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.559238 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:23:59.561057 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.561043 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" Jul 06 22:23:59.567484 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.567470 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 22:23:59.572606 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.572590 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 22:23:59.572711 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.572693 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:23:59.572760 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.572701 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 22:23:59.589255 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.589237 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:23:59.613199 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.613171 2564 apiserver.go:52] "Watching apiserver" Jul 06 22:23:59.620145 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.619973 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 22:23:59.620321 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.620304 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-qb6jc","kube-system/konnectivity-agent-6kkfd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7","openshift-image-registry/node-ca-g8jwm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal","openshift-multus/multus-5j788","openshift-multus/multus-additional-cni-plugins-zkjhk","openshift-multus/network-metrics-daemon-dm8zz","kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xcpg5","openshift-dns/node-resolver-nsz97","openshift-network-diagnostics/network-check-target-j25q7","openshift-network-operator/iptables-alerter-cs9sl","openshift-ovn-kubernetes/ovnkube-node-4fhw9"] Jul 06 22:23:59.621807 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.621790 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.622821 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.622806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.624261 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.624171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.624307 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.624477 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.624818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-m2klg\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.624936 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.624948 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.625210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.625518 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.625578 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.625810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.625680 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cbd6w\"" Jul 06 22:23:59.626575 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.626413 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 22:23:59.626647 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.626625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.626942 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.626887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.627249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.627225 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-h7vc2\"" Jul 06 22:23:59.627437 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.627406 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.628560 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.628545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.629053 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.629020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.629130 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.629092 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 22:23:59.629130 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.629095 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j7dvx\"" Jul 06 22:23:59.629284 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.629273 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.629618 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.629607 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5j788" Jul 06 22:23:59.630710 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.630698 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:23:59.630792 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.630744 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:23:59.631656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631635 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 22:23:59.631656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631648 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 22:23:59.631799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631660 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.631799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631687 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 22:23:59.631799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631659 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 22:23:59.631799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631741 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 22:23:59.631799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631752 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.631962 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-52nh2\"" Jul 06 22:23:59.631994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.631968 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.632127 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.632115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rhmpk\"" Jul 06 22:23:59.633394 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.633380 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.633960 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.633944 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rbsjk\"" Jul 06 22:23:59.634059 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.633973 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.634059 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.633981 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.634159 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.634148 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 22:23:59.635139 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.635120 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.635371 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.635354 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 22:23:59.635626 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.635612 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 22:23:59.635734 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.635712 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ghpb2\"" Jul 06 22:23:59.636751 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.636737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.636819 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.636793 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:23:59.637264 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.637248 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 22:23:59.637354 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.637250 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:23:59.637354 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.637314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kdvfk\"" Jul 06 22:23:59.638127 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.638111 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:23:59.638207 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.638165 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:23:59.641238 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.641224 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 22:23:59.641307 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.641240 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 22:18:58 +0000 UTC" deadline="2028-01-29 00:47:31.581335728 +0000 UTC" Jul 06 22:23:59.641307 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.641281 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13706h23m31.940056591s" Jul 06 22:23:59.641963 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.641947 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 22:23:59.652527 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.652512 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 22:23:59.655954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.655938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a70d449-b56c-4f5b-8af3-90734fa84198-kubelet-config\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.656027 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.655959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-multus-certs\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656027 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.655976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-modprobe-d\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.656027 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.655989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-systemd\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.656027 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caaf6673-9f01-40c6-a0ee-fe4992b9385a-tmp-dir\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.656172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0827b7d5-e529-4e47-9882-b207e32dc594-ovn-node-metrics-cert\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-k8s-cni-cncf-io\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cnibin\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656253 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnhr\" (UniqueName: \"kubernetes.io/projected/b0aa773f-d298-4bde-a4fb-95518c2339ac-kube-api-access-mjnhr\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656280 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-socket-dir-parent\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ggzl\" (UniqueName: \"kubernetes.io/projected/2172046f-3f2b-40c3-be14-dac49e1160bd-kube-api-access-6ggzl\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94be3bea-576a-445b-ae39-58e3dff104de-agent-certs\") pod \"konnectivity-agent-6kkfd\" (UID: \"94be3bea-576a-445b-ae39-58e3dff104de\") " pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-run\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-sys\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.656398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f294w\" (UniqueName: \"kubernetes.io/projected/caaf6673-9f01-40c6-a0ee-fe4992b9385a-kube-api-access-f294w\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-systemd-units\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656471 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-ovnkube-config\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-cnibin\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656544 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysctl-conf\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-kubelet\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-env-overrides\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-ovnkube-script-lib\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656660 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656686 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-system-cni-dir\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-netns\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-cni-multus\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.656785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-daemon-config\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-run-netns\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2172046f-3f2b-40c3-be14-dac49e1160bd-cni-binary-copy\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656890 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-lib-modules\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-var-lib-kubelet\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nphqg\" (UniqueName: \"kubernetes.io/projected/b3083d79-4ef6-4fae-9cb5-716cf551532c-kube-api-access-nphqg\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.656987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-conf-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657024 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-kubernetes\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-host\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-etc-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657076 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8krs\" (UniqueName: \"kubernetes.io/projected/0827b7d5-e529-4e47-9882-b207e32dc594-kube-api-access-m8krs\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b479c6ce-c0e5-443e-a643-a88c73931e4b-iptables-alerter-script\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657109 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b479c6ce-c0e5-443e-a643-a88c73931e4b-host-slash\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkctf\" (UniqueName: \"kubernetes.io/projected/b479c6ce-c0e5-443e-a643-a88c73931e4b-kube-api-access-rkctf\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/caaf6673-9f01-40c6-a0ee-fe4992b9385a-hosts-file\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.657228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94be3bea-576a-445b-ae39-58e3dff104de-konnectivity-ca\") pod \"konnectivity-agent-6kkfd\" (UID: \"94be3bea-576a-445b-ae39-58e3dff104de\") " pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysconfig\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-log-socket\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-os-release\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-device-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0aa773f-d298-4bde-a4fb-95518c2339ac-host\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657275 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-var-lib-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-ovn\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-node-log\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-cni-netd\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccq25\" (UniqueName: \"kubernetes.io/projected/3c00558f-c19d-4d65-82c8-73dd2d6d094e-kube-api-access-ccq25\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-socket-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lnz\" (UniqueName: \"kubernetes.io/projected/d07aafbb-f201-478d-957d-dd33e8c372e5-kube-api-access-m2lnz\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657383 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a70d449-b56c-4f5b-8af3-90734fa84198-dbus\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.657780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657413 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d07aafbb-f201-478d-957d-dd33e8c372e5-tmp\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-os-release\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-cni-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-cni-bin\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657505 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-kubelet\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-hostroot\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-etc-kubernetes\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657590 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysctl-d\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-tuned\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-slash\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657659 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657683 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-registration-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-sys-fs\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0aa773f-d298-4bde-a4fb-95518c2339ac-serviceca\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.658186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdphn\" (UniqueName: \"kubernetes.io/projected/a541e958-955c-43a7-9572-845dfd6637ce-kube-api-access-mdphn\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:23:59.658656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.658656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657832 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-system-cni-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.658656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-systemd\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.658656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.657873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-cni-bin\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.670891 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.670876 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-h8jjt" Jul 06 22:23:59.676604 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.676591 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-h8jjt" Jul 06 22:23:59.758962 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.758941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b479c6ce-c0e5-443e-a643-a88c73931e4b-host-slash\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.758966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkctf\" (UniqueName: \"kubernetes.io/projected/b479c6ce-c0e5-443e-a643-a88c73931e4b-kube-api-access-rkctf\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.758980 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/caaf6673-9f01-40c6-a0ee-fe4992b9385a-hosts-file\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.758994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94be3bea-576a-445b-ae39-58e3dff104de-konnectivity-ca\") pod \"konnectivity-agent-6kkfd\" (UID: \"94be3bea-576a-445b-ae39-58e3dff104de\") " pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysconfig\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-log-socket\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759068 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-log-socket\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759068 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysconfig\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-os-release\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759116 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b479c6ce-c0e5-443e-a643-a88c73931e4b-host-slash\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759145 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/caaf6673-9f01-40c6-a0ee-fe4992b9385a-hosts-file\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759175 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-device-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759198 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-os-release\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0aa773f-d298-4bde-a4fb-95518c2339ac-host\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-var-lib-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-ovn\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759259 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-device-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-node-log\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759307 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-ovn\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759308 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-var-lib-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-cni-netd\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759352 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b0aa773f-d298-4bde-a4fb-95518c2339ac-host\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccq25\" (UniqueName: \"kubernetes.io/projected/3c00558f-c19d-4d65-82c8-73dd2d6d094e-kube-api-access-ccq25\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-socket-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.759411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759388 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-cni-netd\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lnz\" (UniqueName: \"kubernetes.io/projected/d07aafbb-f201-478d-957d-dd33e8c372e5-kube-api-access-m2lnz\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759447 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-node-log\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759487 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a70d449-b56c-4f5b-8af3-90734fa84198-dbus\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759489 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-run-ovn-kubernetes\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759511 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-socket-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759619 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/94be3bea-576a-445b-ae39-58e3dff104de-konnectivity-ca\") pod \"konnectivity-agent-6kkfd\" (UID: \"94be3bea-576a-445b-ae39-58e3dff104de\") " pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d07aafbb-f201-478d-957d-dd33e8c372e5-tmp\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759637 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a70d449-b56c-4f5b-8af3-90734fa84198-dbus\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-os-release\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-cni-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759700 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-cni-bin\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759709 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759721 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-os-release\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-kubelet\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-hostroot\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759763 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-cni-bin\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759779 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-cni-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-etc-kubernetes\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759782 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-kubelet\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759802 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-hostroot\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysctl-d\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-tuned\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759831 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-etc-kubernetes\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-slash\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759895 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysctl-d\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759887 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759937 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759936 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-slash\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-registration-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.759977 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.759995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-sys-fs\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.760952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760024 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-registration-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.760045 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:00.26002388 +0000 UTC m=+1.996425874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760072 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-sys-fs\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0aa773f-d298-4bde-a4fb-95518c2339ac-serviceca\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760142 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdphn\" (UniqueName: \"kubernetes.io/projected/a541e958-955c-43a7-9572-845dfd6637ce-kube-api-access-mdphn\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-system-cni-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-systemd\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-cni-bin\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a70d449-b56c-4f5b-8af3-90734fa84198-kubelet-config\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760262 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-etc-selinux\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-multus-certs\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760286 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-run-systemd\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-modprobe-d\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-systemd\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760344 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a70d449-b56c-4f5b-8af3-90734fa84198-kubelet-config\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760346 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-system-cni-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.761791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760306 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-cni-bin\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caaf6673-9f01-40c6-a0ee-fe4992b9385a-tmp-dir\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760363 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-multus-certs\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760402 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-systemd\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760440 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760446 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-modprobe-d\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760476 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0827b7d5-e529-4e47-9882-b207e32dc594-ovn-node-metrics-cert\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760476 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760499 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cni-binary-copy\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760576 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-k8s-cni-cncf-io\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cnibin\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760658 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-cnibin\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760657 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b0aa773f-d298-4bde-a4fb-95518c2339ac-serviceca\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760665 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caaf6673-9f01-40c6-a0ee-fe4992b9385a-tmp-dir\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760658 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-k8s-cni-cncf-io\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760682 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnhr\" (UniqueName: \"kubernetes.io/projected/b0aa773f-d298-4bde-a4fb-95518c2339ac-kube-api-access-mjnhr\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.762572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760704 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b3083d79-4ef6-4fae-9cb5-716cf551532c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760719 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-socket-dir-parent\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ggzl\" (UniqueName: \"kubernetes.io/projected/2172046f-3f2b-40c3-be14-dac49e1160bd-kube-api-access-6ggzl\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760755 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-socket-dir-parent\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760760 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94be3bea-576a-445b-ae39-58e3dff104de-agent-certs\") pod \"konnectivity-agent-6kkfd\" (UID: \"94be3bea-576a-445b-ae39-58e3dff104de\") " pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-run\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-sys\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f294w\" (UniqueName: \"kubernetes.io/projected/caaf6673-9f01-40c6-a0ee-fe4992b9385a-kube-api-access-f294w\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-systemd-units\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760886 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-run\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-ovnkube-config\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-cnibin\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760946 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-sys\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysctl-conf\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.760981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-kubelet\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-env-overrides\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-ovnkube-script-lib\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-system-cni-dir\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761115 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-netns\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-cni-multus\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-daemon-config\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761205 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-run-netns\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2172046f-3f2b-40c3-be14-dac49e1160bd-cni-binary-copy\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-lib-modules\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-var-lib-kubelet\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nphqg\" (UniqueName: \"kubernetes.io/projected/b3083d79-4ef6-4fae-9cb5-716cf551532c-kube-api-access-nphqg\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-conf-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-kubernetes\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-host\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761469 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-ovnkube-config\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-etc-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.763808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761471 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3c00558f-c19d-4d65-82c8-73dd2d6d094e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761525 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-systemd-units\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8krs\" (UniqueName: \"kubernetes.io/projected/0827b7d5-e529-4e47-9882-b207e32dc594-kube-api-access-m8krs\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761565 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b479c6ce-c0e5-443e-a643-a88c73931e4b-iptables-alerter-script\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761753 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-run-netns\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762017 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-daemon-config\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762046 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b479c6ce-c0e5-443e-a643-a88c73931e4b-iptables-alerter-script\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.761536 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-cnibin\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762066 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-host-kubelet\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762227 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2172046f-3f2b-40c3-be14-dac49e1160bd-cni-binary-copy\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762299 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-kubernetes\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762347 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-lib-modules\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762358 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-host\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762378 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-env-overrides\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-multus-conf-dir\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762399 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0827b7d5-e529-4e47-9882-b207e32dc594-etc-openvswitch\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762461 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-var-lib-kubelet\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762576 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.764246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762831 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-sysctl-conf\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762839 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0827b7d5-e529-4e47-9882-b207e32dc594-ovnkube-script-lib\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762875 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3c00558f-c19d-4d65-82c8-73dd2d6d094e-system-cni-dir\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.762889 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.763216 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d07aafbb-f201-478d-957d-dd33e8c372e5-tmp\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762933 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-var-lib-cni-multus\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.762902 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2172046f-3f2b-40c3-be14-dac49e1160bd-host-run-netns\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.763260 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:24:00.263243216 +0000 UTC m=+1.999645192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.763280 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0827b7d5-e529-4e47-9882-b207e32dc594-ovn-node-metrics-cert\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.763405 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d07aafbb-f201-478d-957d-dd33e8c372e5-etc-tuned\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.764921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.763411 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/94be3bea-576a-445b-ae39-58e3dff104de-agent-certs\") pod \"konnectivity-agent-6kkfd\" (UID: \"94be3bea-576a-445b-ae39-58e3dff104de\") " pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.768411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.768034 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkctf\" (UniqueName: \"kubernetes.io/projected/b479c6ce-c0e5-443e-a643-a88c73931e4b-kube-api-access-rkctf\") pod \"iptables-alerter-cs9sl\" (UID: \"b479c6ce-c0e5-443e-a643-a88c73931e4b\") " pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:23:59.768411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.768038 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccq25\" (UniqueName: \"kubernetes.io/projected/3c00558f-c19d-4d65-82c8-73dd2d6d094e-kube-api-access-ccq25\") pod \"multus-additional-cni-plugins-zkjhk\" (UID: \"3c00558f-c19d-4d65-82c8-73dd2d6d094e\") " pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.768411 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.768406 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:23:59.768630 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.768449 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:23:59.768630 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.768462 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:23:59.768630 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:23:59.768514 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:00.268497905 +0000 UTC m=+2.004899868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:23:59.770241 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.770216 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f294w\" (UniqueName: \"kubernetes.io/projected/caaf6673-9f01-40c6-a0ee-fe4992b9385a-kube-api-access-f294w\") pod \"node-resolver-nsz97\" (UID: \"caaf6673-9f01-40c6-a0ee-fe4992b9385a\") " pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.770477 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.770457 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nphqg\" (UniqueName: \"kubernetes.io/projected/b3083d79-4ef6-4fae-9cb5-716cf551532c-kube-api-access-nphqg\") pod \"aws-ebs-csi-driver-node-5mnd7\" (UID: \"b3083d79-4ef6-4fae-9cb5-716cf551532c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.770807 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.770779 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ggzl\" (UniqueName: \"kubernetes.io/projected/2172046f-3f2b-40c3-be14-dac49e1160bd-kube-api-access-6ggzl\") pod \"multus-5j788\" (UID: \"2172046f-3f2b-40c3-be14-dac49e1160bd\") " pod="openshift-multus/multus-5j788" Jul 06 22:23:59.770892 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.770860 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lnz\" (UniqueName: \"kubernetes.io/projected/d07aafbb-f201-478d-957d-dd33e8c372e5-kube-api-access-m2lnz\") pod \"tuned-xcpg5\" (UID: \"d07aafbb-f201-478d-957d-dd33e8c372e5\") " pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.771049 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.771027 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8krs\" (UniqueName: \"kubernetes.io/projected/0827b7d5-e529-4e47-9882-b207e32dc594-kube-api-access-m8krs\") pod \"ovnkube-node-4fhw9\" (UID: \"0827b7d5-e529-4e47-9882-b207e32dc594\") " pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:23:59.771087 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.771064 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdphn\" (UniqueName: \"kubernetes.io/projected/a541e958-955c-43a7-9572-845dfd6637ce-kube-api-access-mdphn\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:23:59.771087 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.771077 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnhr\" (UniqueName: \"kubernetes.io/projected/b0aa773f-d298-4bde-a4fb-95518c2339ac-kube-api-access-mjnhr\") pod \"node-ca-g8jwm\" (UID: \"b0aa773f-d298-4bde-a4fb-95518c2339ac\") " pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:23:59.783004 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.782989 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:23:59.784015 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.783999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" Jul 06 22:23:59.831845 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:59.831808 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86df2e047708b9ac3980718e830e2fcf.slice/crio-4ebea33d7cd9a6b590963c4621bb66159281a057c1b1cdc3369bf99846ad6714 WatchSource:0}: Error finding container 4ebea33d7cd9a6b590963c4621bb66159281a057c1b1cdc3369bf99846ad6714: Status 404 returned error can't find the container with id 4ebea33d7cd9a6b590963c4621bb66159281a057c1b1cdc3369bf99846ad6714 Jul 06 22:23:59.832231 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:59.832215 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6383cbabd87298deaa3c473fa7d5636c.slice/crio-9faac2b44ef4edc962f26ee32bcd816b22979174db3df9506e8a748278637407 WatchSource:0}: Error finding container 9faac2b44ef4edc962f26ee32bcd816b22979174db3df9506e8a748278637407: Status 404 returned error can't find the container with id 9faac2b44ef4edc962f26ee32bcd816b22979174db3df9506e8a748278637407 Jul 06 22:23:59.837008 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.836991 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:23:59.955458 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.955397 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" Jul 06 22:23:59.963496 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:59.963473 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c00558f_c19d_4d65_82c8_73dd2d6d094e.slice/crio-08d34047ca8f70a1729f3986f72cc2113ab3823dbacb2e3c65523a9ab056e9c2 WatchSource:0}: Error finding container 08d34047ca8f70a1729f3986f72cc2113ab3823dbacb2e3c65523a9ab056e9c2: Status 404 returned error can't find the container with id 08d34047ca8f70a1729f3986f72cc2113ab3823dbacb2e3c65523a9ab056e9c2 Jul 06 22:23:59.968548 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.968530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nsz97" Jul 06 22:23:59.975369 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:59.975348 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaaf6673_9f01_40c6_a0ee_fe4992b9385a.slice/crio-8809adc27bd94dddfe3f357ff665b9fba38bafe07daf4701b303d6608e1f9672 WatchSource:0}: Error finding container 8809adc27bd94dddfe3f357ff665b9fba38bafe07daf4701b303d6608e1f9672: Status 404 returned error can't find the container with id 8809adc27bd94dddfe3f357ff665b9fba38bafe07daf4701b303d6608e1f9672 Jul 06 22:23:59.984878 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.984861 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" Jul 06 22:23:59.990994 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:23:59.990971 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3083d79_4ef6_4fae_9cb5_716cf551532c.slice/crio-3348ae2efcce88b4c62d9daf8ccfda2469d4f2898bcffc1b62f2d3cafff23e6a WatchSource:0}: Error finding container 3348ae2efcce88b4c62d9daf8ccfda2469d4f2898bcffc1b62f2d3cafff23e6a: Status 404 returned error can't find the container with id 3348ae2efcce88b4c62d9daf8ccfda2469d4f2898bcffc1b62f2d3cafff23e6a Jul 06 22:23:59.997991 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:23:59.997976 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g8jwm" Jul 06 22:24:00.002666 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.002652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:00.003885 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:00.003868 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0aa773f_d298_4bde_a4fb_95518c2339ac.slice/crio-1f8928d2550ec6f5e361c924019b10f40806afa4d0f33b12fb03fa6340ee1ac1 WatchSource:0}: Error finding container 1f8928d2550ec6f5e361c924019b10f40806afa4d0f33b12fb03fa6340ee1ac1: Status 404 returned error can't find the container with id 1f8928d2550ec6f5e361c924019b10f40806afa4d0f33b12fb03fa6340ee1ac1 Jul 06 22:24:00.009036 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:00.009018 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0827b7d5_e529_4e47_9882_b207e32dc594.slice/crio-da9729f1f8ef5221f3c82e7177b09e96b906e90e1e85ee85ed669d5b097fee4a WatchSource:0}: Error finding container da9729f1f8ef5221f3c82e7177b09e96b906e90e1e85ee85ed669d5b097fee4a: Status 404 returned error can't find the container with id da9729f1f8ef5221f3c82e7177b09e96b906e90e1e85ee85ed669d5b097fee4a Jul 06 22:24:00.017850 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.017833 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5j788" Jul 06 22:24:00.024436 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:00.024403 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2172046f_3f2b_40c3_be14_dac49e1160bd.slice/crio-e55198f74fb7c12d6cf48616f2862f76405582824e40bc55db9241fbae1ee2c1 WatchSource:0}: Error finding container e55198f74fb7c12d6cf48616f2862f76405582824e40bc55db9241fbae1ee2c1: Status 404 returned error can't find the container with id e55198f74fb7c12d6cf48616f2862f76405582824e40bc55db9241fbae1ee2c1 Jul 06 22:24:00.045321 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.045292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cs9sl" Jul 06 22:24:00.051329 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:00.051312 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb479c6ce_c0e5_443e_a643_a88c73931e4b.slice/crio-977de7c2a0d1f330e046e0700c583d38b11e08e9ab061b626e1f0878b9752bc0 WatchSource:0}: Error finding container 977de7c2a0d1f330e046e0700c583d38b11e08e9ab061b626e1f0878b9752bc0: Status 404 returned error can't find the container with id 977de7c2a0d1f330e046e0700c583d38b11e08e9ab061b626e1f0878b9752bc0 Jul 06 22:24:00.091684 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:00.091660 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94be3bea_576a_445b_ae39_58e3dff104de.slice/crio-b342a934f7e51dc3715cf584ff7747917663e09ed50c3ab2f715c8767570b3b2 WatchSource:0}: Error finding container b342a934f7e51dc3715cf584ff7747917663e09ed50c3ab2f715c8767570b3b2: Status 404 returned error can't find the container with id b342a934f7e51dc3715cf584ff7747917663e09ed50c3ab2f715c8767570b3b2 Jul 06 22:24:00.101486 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:00.101466 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07aafbb_f201_478d_957d_dd33e8c372e5.slice/crio-01cad6369a22d04aef4a15f11c17e939153cba972ba2f1fd6ea94d01dcdc1257 WatchSource:0}: Error finding container 01cad6369a22d04aef4a15f11c17e939153cba972ba2f1fd6ea94d01dcdc1257: Status 404 returned error can't find the container with id 01cad6369a22d04aef4a15f11c17e939153cba972ba2f1fd6ea94d01dcdc1257 Jul 06 22:24:00.265212 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.265148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:00.265212 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.265195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:00.265398 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.265291 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:00.265398 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.265350 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:01.265336327 +0000 UTC m=+3.001738293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:00.265398 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.265291 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:00.265543 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.265414 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:24:01.265402979 +0000 UTC m=+3.001804961 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:00.367163 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.366598 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:00.367163 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.366778 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:00.367163 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.366798 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:00.367163 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.366810 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:00.367163 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.366862 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:01.36684619 +0000 UTC m=+3.103248172 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:00.678289 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.678191 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:18:59 +0000 UTC" deadline="2027-12-22 06:48:07.65923784 +0000 UTC" Jul 06 22:24:00.678289 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.678241 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12800h24m6.981001994s" Jul 06 22:24:00.804107 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.804064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:00.804263 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.804210 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:00.804618 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.804595 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:00.804741 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.804717 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:00.804874 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.804815 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:00.804993 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:00.804972 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:00.846462 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.846384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" event={"ID":"d07aafbb-f201-478d-957d-dd33e8c372e5","Type":"ContainerStarted","Data":"01cad6369a22d04aef4a15f11c17e939153cba972ba2f1fd6ea94d01dcdc1257"} Jul 06 22:24:00.859273 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.859241 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6kkfd" event={"ID":"94be3bea-576a-445b-ae39-58e3dff104de","Type":"ContainerStarted","Data":"b342a934f7e51dc3715cf584ff7747917663e09ed50c3ab2f715c8767570b3b2"} Jul 06 22:24:00.887538 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.887447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cs9sl" event={"ID":"b479c6ce-c0e5-443e-a643-a88c73931e4b","Type":"ContainerStarted","Data":"977de7c2a0d1f330e046e0700c583d38b11e08e9ab061b626e1f0878b9752bc0"} Jul 06 22:24:00.912051 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.912014 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5j788" event={"ID":"2172046f-3f2b-40c3-be14-dac49e1160bd","Type":"ContainerStarted","Data":"e55198f74fb7c12d6cf48616f2862f76405582824e40bc55db9241fbae1ee2c1"} Jul 06 22:24:00.920784 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.920748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8jwm" event={"ID":"b0aa773f-d298-4bde-a4fb-95518c2339ac","Type":"ContainerStarted","Data":"1f8928d2550ec6f5e361c924019b10f40806afa4d0f33b12fb03fa6340ee1ac1"} Jul 06 22:24:00.933438 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.933345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerStarted","Data":"08d34047ca8f70a1729f3986f72cc2113ab3823dbacb2e3c65523a9ab056e9c2"} Jul 06 22:24:00.959379 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.959345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" event={"ID":"86df2e047708b9ac3980718e830e2fcf","Type":"ContainerStarted","Data":"4ebea33d7cd9a6b590963c4621bb66159281a057c1b1cdc3369bf99846ad6714"} Jul 06 22:24:00.963777 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.963663 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"da9729f1f8ef5221f3c82e7177b09e96b906e90e1e85ee85ed669d5b097fee4a"} Jul 06 22:24:00.969477 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.969454 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:00.995926 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:00.995710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" event={"ID":"b3083d79-4ef6-4fae-9cb5-716cf551532c","Type":"ContainerStarted","Data":"3348ae2efcce88b4c62d9daf8ccfda2469d4f2898bcffc1b62f2d3cafff23e6a"} Jul 06 22:24:01.014367 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.014335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nsz97" event={"ID":"caaf6673-9f01-40c6-a0ee-fe4992b9385a","Type":"ContainerStarted","Data":"8809adc27bd94dddfe3f357ff665b9fba38bafe07daf4701b303d6608e1f9672"} Jul 06 22:24:01.019132 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.019079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" event={"ID":"6383cbabd87298deaa3c473fa7d5636c","Type":"ContainerStarted","Data":"9faac2b44ef4edc962f26ee32bcd816b22979174db3df9506e8a748278637407"} Jul 06 22:24:01.086474 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.086352 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 22:24:01.273746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.273672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:01.273905 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.273751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:01.273905 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.273883 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:01.274012 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.273940 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:03.273922287 +0000 UTC m=+5.010324255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:01.274327 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.274304 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:01.274444 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.274361 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:24:03.27434474 +0000 UTC m=+5.010746709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:01.376111 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.376069 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:01.376286 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.376227 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:01.376286 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.376247 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:01.376286 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.376260 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:01.376454 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:01.376315 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:03.376296285 +0000 UTC m=+5.112698250 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:01.683007 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.682901 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 22:18:59 +0000 UTC" deadline="2027-12-09 22:11:56.127755732 +0000 UTC" Jul 06 22:24:01.683007 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:01.682963 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12503h47m54.44479689s" Jul 06 22:24:02.806964 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:02.806936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:02.807454 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:02.807077 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:02.807529 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:02.807467 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:02.807584 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:02.807567 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:02.807633 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:02.807613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:02.807692 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:02.807674 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:03.295635 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:03.294767 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:03.295635 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:03.294847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:03.295635 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.294998 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:03.295635 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.295066 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:24:07.295046134 +0000 UTC m=+9.031448112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:03.295635 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.295474 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:03.295635 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.295529 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:07.295514273 +0000 UTC m=+9.031916257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:03.395716 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:03.395679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:03.395895 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.395841 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:03.395895 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.395861 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:03.395895 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.395872 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:03.396042 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:03.395930 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:07.39591163 +0000 UTC m=+9.132313614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:04.804539 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:04.804417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:04.805022 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:04.804585 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:04.805022 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:04.804628 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:04.805022 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:04.804701 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:04.805523 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:04.805286 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:04.805523 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:04.805405 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:06.804560 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:06.804523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:06.805024 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:06.804662 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:06.805024 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:06.804727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:06.805024 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:06.804802 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:06.805024 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:06.804489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:06.805230 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:06.805088 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:07.328615 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:07.328575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:07.328777 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:07.328693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:07.328816 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.328803 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:07.329212 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.328891 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:15.32884432 +0000 UTC m=+17.065246291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:07.329212 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.328958 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:07.329212 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.329023 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:24:15.329008243 +0000 UTC m=+17.065410217 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:07.429546 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:07.429399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:07.429729 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.429634 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:07.429729 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.429658 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:07.429729 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.429672 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:07.429888 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:07.429736 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:15.42971634 +0000 UTC m=+17.166118322 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:08.808347 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:08.808309 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:08.808812 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:08.808457 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:08.808812 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:08.808749 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:08.808908 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:08.808879 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:08.808953 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:08.808915 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:08.809022 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:08.808998 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:10.804613 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:10.804579 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:10.804613 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:10.804608 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:10.805097 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:10.804576 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:10.805097 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:10.804701 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:10.805097 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:10.804850 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:10.805097 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:10.804981 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:12.804241 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:12.804203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:12.804718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:12.804209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:12.804718 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:12.804330 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:12.804718 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:12.804436 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:12.804718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:12.804209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:12.804718 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:12.804516 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:14.804222 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:14.803999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:14.804676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:14.803999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:14.804676 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:14.804328 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:14.804676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:14.804058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:14.804676 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:14.804455 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:14.804676 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:14.804555 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:15.393669 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:15.393637 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:15.393825 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:15.393699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:15.393825 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.393792 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:15.393905 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.393855 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:24:31.393839404 +0000 UTC m=+33.130241372 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:15.393905 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.393792 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:15.393977 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.393916 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:31.393904896 +0000 UTC m=+33.130306865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:15.494479 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:15.494449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:15.494645 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.494624 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:15.494699 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.494655 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:15.494699 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.494668 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:15.494762 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:15.494725 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:31.494711586 +0000 UTC m=+33.231113551 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:16.804413 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:16.804375 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:16.804923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:16.804388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:16.804923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:16.804389 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:16.804923 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:16.804533 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:16.804923 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:16.804644 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:16.804923 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:16.804750 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:18.805419 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:18.805383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:18.805868 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:18.805505 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:18.805868 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:18.805525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:18.805868 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:18.805665 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:18.805868 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:18.805687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:18.805868 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:18.805809 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:20.060300 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.060274 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" event={"ID":"d07aafbb-f201-478d-957d-dd33e8c372e5","Type":"ContainerStarted","Data":"09daeefa0de075633373259a126aacbb98a40496fef096b86adce209d2812f12"} Jul 06 22:24:20.061756 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.061734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" event={"ID":"86df2e047708b9ac3980718e830e2fcf","Type":"ContainerStarted","Data":"90f728888ed6386dc5aa3539852b718a8315acb3818e60ed39f603950c640ea3"} Jul 06 22:24:20.065670 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.065645 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"78f7c9b936435591c7eee2c5721f08759248c753800258178f42d464c64ab309"} Jul 06 22:24:20.076306 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.076272 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xcpg5" podStartSLOduration=2.365091453 podStartE2EDuration="22.076260565s" totalImagesPullingTime="19.711169112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:20.076004774 +0000 UTC m=+21.812406760" watchObservedRunningTime="2026-07-06 22:24:20.076260565 +0000 UTC m=+21.812662551" Jul 06 22:24:20.805078 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.804742 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:20.805206 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.804757 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:20.805206 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:20.805131 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:20.805206 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:20.804808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:20.805364 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:20.805232 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:20.805364 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:20.805305 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:21.070531 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.070442 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:24:21.071239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.070861 2564 generic.go:356] "Generic (PLEG): container finished" podID="0827b7d5-e529-4e47-9882-b207e32dc594" containerID="10725065d8d7fc61e055303cce20a0099eadaa1407250dd4b92dc20436cf508a" exitCode=1 Jul 06 22:24:21.071239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.070933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"ce12bf53f47107fb6fd7ff82664c1652473e92df7efacfb5522aacc81de96c5b"} Jul 06 22:24:21.071239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.070994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"62e97223960ed7c2692035f127620f8f00b9c7cbef0112389f82302803cb722d"} Jul 06 22:24:21.071239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.071037 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"6ddcb3e3df4547c15a53a1dcf4f8fec45efdaee40b4db57f0c22c8b76ff85dc8"} Jul 06 22:24:21.071239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.071078 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"c64c1e681fd9f2b520c157b66a30a2f305e1aec1dbbaa1b40ef83067fea507b6"} Jul 06 22:24:21.071239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.071152 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerDied","Data":"10725065d8d7fc61e055303cce20a0099eadaa1407250dd4b92dc20436cf508a"} Jul 06 22:24:21.072314 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.072294 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" event={"ID":"b3083d79-4ef6-4fae-9cb5-716cf551532c","Type":"ContainerStarted","Data":"b5d9a44ecd1491a947effb50505f8f9d6abaa51e7fb01fca14159f09eeb95ba3"} Jul 06 22:24:21.073670 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.073641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nsz97" event={"ID":"caaf6673-9f01-40c6-a0ee-fe4992b9385a","Type":"ContainerStarted","Data":"52586ae0df6c3d66ebbf0bbec55dc3b3a30967fbdaad524044b3b52b82c1f7e7"} Jul 06 22:24:21.075043 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.075022 2564 generic.go:356] "Generic (PLEG): container finished" podID="6383cbabd87298deaa3c473fa7d5636c" containerID="ede7b16ca3f68d1b104546df4e7f66557d55066283f5e5a63b5b82dc3d784e46" exitCode=0 Jul 06 22:24:21.075141 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.075085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" event={"ID":"6383cbabd87298deaa3c473fa7d5636c","Type":"ContainerDied","Data":"ede7b16ca3f68d1b104546df4e7f66557d55066283f5e5a63b5b82dc3d784e46"} Jul 06 22:24:21.076541 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.076496 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6kkfd" event={"ID":"94be3bea-576a-445b-ae39-58e3dff104de","Type":"ContainerStarted","Data":"ed59356ed08e53e7f737d969064a126f85818542986ab6fc1a9e33fcd36e6a5f"} Jul 06 22:24:21.081182 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.081159 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5j788" event={"ID":"2172046f-3f2b-40c3-be14-dac49e1160bd","Type":"ContainerStarted","Data":"95cfcf65959a466c0d9f2c186e1b8a90cef9280ec7410597360c46d8a569f144"} Jul 06 22:24:21.082534 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.082513 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g8jwm" event={"ID":"b0aa773f-d298-4bde-a4fb-95518c2339ac","Type":"ContainerStarted","Data":"9010d9bd3255971578febbd94a2b3e1a4e7058b844ab53a892cd60d7cb3d7d58"} Jul 06 22:24:21.083947 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.083927 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c00558f-c19d-4d65-82c8-73dd2d6d094e" containerID="9a6ae4ee1a1d183cf3f1b686e2c069ff44f9ddc64d8c8b078901c323909a31cc" exitCode=0 Jul 06 22:24:21.084030 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.083995 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerDied","Data":"9a6ae4ee1a1d183cf3f1b686e2c069ff44f9ddc64d8c8b078901c323909a31cc"} Jul 06 22:24:21.088716 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.088681 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-55.ec2.internal" podStartSLOduration=22.088667374 podStartE2EDuration="22.088667374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:20.08926427 +0000 UTC m=+21.825666259" watchObservedRunningTime="2026-07-06 22:24:21.088667374 +0000 UTC m=+22.825069363" Jul 06 22:24:21.088816 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.088788 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nsz97" podStartSLOduration=3.25203244 podStartE2EDuration="23.088782078s" totalImagesPullingTime="19.836749638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:21.08850981 +0000 UTC m=+22.824911801" watchObservedRunningTime="2026-07-06 22:24:21.088782078 +0000 UTC m=+22.825184066" Jul 06 22:24:21.105681 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.105647 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5j788" podStartSLOduration=2.989503651 podStartE2EDuration="23.105635342s" totalImagesPullingTime="20.116131691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:21.105061548 +0000 UTC m=+22.841463537" watchObservedRunningTime="2026-07-06 22:24:21.105635342 +0000 UTC m=+22.842037328" Jul 06 22:24:21.139738 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.139698 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6kkfd" podStartSLOduration=3.453858333 podStartE2EDuration="23.139684544s" totalImagesPullingTime="19.685826211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:21.139096704 +0000 UTC m=+22.875498690" watchObservedRunningTime="2026-07-06 22:24:21.139684544 +0000 UTC m=+22.876086532" Jul 06 22:24:21.173460 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.173405 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g8jwm" podStartSLOduration=3.385346808 podStartE2EDuration="23.173393824s" totalImagesPullingTime="19.788047016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:21.17311946 +0000 UTC m=+22.909521460" watchObservedRunningTime="2026-07-06 22:24:21.173393824 +0000 UTC m=+22.909795810" Jul 06 22:24:21.611240 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.611208 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:24:21.611759 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.611740 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:24:21.648100 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.648069 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:24:21.648611 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:21.648590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6kkfd" Jul 06 22:24:22.083459 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.083438 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 22:24:22.091212 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.091182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cs9sl" event={"ID":"b479c6ce-c0e5-443e-a643-a88c73931e4b","Type":"ContainerStarted","Data":"78bd599ed6f9028a2cffe02ba853463eedae60c32d019ed725fcaa8d92e1ae46"} Jul 06 22:24:22.093466 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.093440 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" event={"ID":"b3083d79-4ef6-4fae-9cb5-716cf551532c","Type":"ContainerStarted","Data":"eeaa980adb91673e3972fc7c05540e52bfda33bfaa2e09512d6a575c6948f010"} Jul 06 22:24:22.095595 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.095575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" event={"ID":"6383cbabd87298deaa3c473fa7d5636c","Type":"ContainerStarted","Data":"3825bb479cd0a55474ded270b515f02d5e1efe9fcc4a3e686dc7dd72d6c40ad2"} Jul 06 22:24:22.118989 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.118948 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cs9sl" podStartSLOduration=4.353737207 podStartE2EDuration="24.118936451s" totalImagesPullingTime="19.765199244s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:22.105371817 +0000 UTC m=+23.841773827" watchObservedRunningTime="2026-07-06 22:24:22.118936451 +0000 UTC m=+23.855338437" Jul 06 22:24:22.119343 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.119308 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-55.ec2.internal" podStartSLOduration=23.119300255 podStartE2EDuration="23.119300255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:22.118712019 +0000 UTC m=+23.855114007" watchObservedRunningTime="2026-07-06 22:24:22.119300255 +0000 UTC m=+23.855702242" Jul 06 22:24:22.804224 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.804197 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:22.804224 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.804211 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:22.804472 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.804210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:22.804472 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:22.804331 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:22.804584 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:22.804482 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:22.804642 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:22.804607 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:22.870090 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.869964 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T22:24:22.083454107Z","UUID":"fe41b176-f482-402a-9d39-0f49fcac3b15","Handler":null,"Name":"","Endpoint":""} Jul 06 22:24:22.872172 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.872148 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 22:24:22.872294 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:22.872195 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 22:24:23.101408 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:23.101349 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:24:23.104048 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:23.104006 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"b2fdfec69e7e233d09daffdb7eca072726a346231c1fddae215168f1ad791faa"} Jul 06 22:24:24.113288 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:24.113254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" event={"ID":"b3083d79-4ef6-4fae-9cb5-716cf551532c","Type":"ContainerStarted","Data":"c761a046c83f7035277a9f0741bdff7882e9b3c2cf3599b570e92c24cd385f48"} Jul 06 22:24:24.133626 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:24.133579 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5mnd7" podStartSLOduration=8.421473853 podStartE2EDuration="26.13356421s" totalImagesPullingTime="17.712090357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:24.133223455 +0000 UTC m=+25.869625448" watchObservedRunningTime="2026-07-06 22:24:24.13356421 +0000 UTC m=+25.869966200" Jul 06 22:24:24.804009 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:24.803805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:24.804173 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:24.804102 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:24.804173 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:24.803870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:24.804280 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:24.804178 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:24.804280 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:24.803814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:24.804280 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:24.804254 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:26.121355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.121297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:24:26.121912 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.121730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"f4419c2e69467ae14c545b975f940cc11ed517f04de99d0b0c25de5f82c204c6"} Jul 06 22:24:26.122056 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.122033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:26.122164 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.122104 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:26.122220 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.122164 2564 scope.go:117] "RemoveContainer" containerID="10725065d8d7fc61e055303cce20a0099eadaa1407250dd4b92dc20436cf508a" Jul 06 22:24:26.123704 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.123679 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c00558f-c19d-4d65-82c8-73dd2d6d094e" containerID="9071cca8ce94d4c48f5d7847ad08185fea0756da9ee9e51d72bcc9f1d3fd4c11" exitCode=0 Jul 06 22:24:26.123789 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.123713 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerDied","Data":"9071cca8ce94d4c48f5d7847ad08185fea0756da9ee9e51d72bcc9f1d3fd4c11"} Jul 06 22:24:26.139678 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.139657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:26.804148 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.803955 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:26.804366 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.803956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:26.804366 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:26.804245 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:26.804366 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:26.803973 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:26.804366 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:26.804314 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:26.804549 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:26.804374 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:27.128468 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.128438 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c00558f-c19d-4d65-82c8-73dd2d6d094e" containerID="79f618f54874a769465019b94abb5feb8d7ef7b6fe710eb3871a3dbf60c99b37" exitCode=0 Jul 06 22:24:27.128894 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.128519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerDied","Data":"79f618f54874a769465019b94abb5feb8d7ef7b6fe710eb3871a3dbf60c99b37"} Jul 06 22:24:27.132508 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.132489 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:24:27.132971 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.132944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" event={"ID":"0827b7d5-e529-4e47-9882-b207e32dc594","Type":"ContainerStarted","Data":"e47b134a463ac8dc04dae9e23f0b05932ee1fd894e244c599907aa8b957fc299"} Jul 06 22:24:27.133335 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.133203 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:27.147485 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.147464 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:27.179637 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.179591 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" podStartSLOduration=9.350684427000001 podStartE2EDuration="29.179576826s" totalImagesPullingTime="19.828892399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:27.179330491 +0000 UTC m=+28.915732479" watchObservedRunningTime="2026-07-06 22:24:27.179576826 +0000 UTC m=+28.915978810" Jul 06 22:24:27.604822 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.604791 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j25q7"] Jul 06 22:24:27.604996 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.604954 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:27.605078 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:27.605054 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:27.605359 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.605342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dm8zz"] Jul 06 22:24:27.605459 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.605448 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:27.605540 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:27.605525 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:27.609253 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.609231 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qb6jc"] Jul 06 22:24:27.609466 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:27.609337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:27.609466 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:27.609414 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:28.376471 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.376439 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:28.506960 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.506933 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:28.607524 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.607496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:28.708351 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.708323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:28.805355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.805330 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:28.805522 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:28.805417 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:28.805522 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.805502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:28.805637 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:28.805611 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:28.805694 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:28.805646 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:28.805743 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:28.805723 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:29.139209 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:29.139133 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c00558f-c19d-4d65-82c8-73dd2d6d094e" containerID="5027de8d446ae7d61278744b8e2df8cf480ec7a5f6c644dd67e586c31ad3587f" exitCode=0 Jul 06 22:24:29.139358 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:29.139204 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerDied","Data":"5027de8d446ae7d61278744b8e2df8cf480ec7a5f6c644dd67e586c31ad3587f"} Jul 06 22:24:30.805029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:30.804371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:30.805029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:30.804376 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:30.805029 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:30.804750 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:30.805029 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:30.804818 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:30.805029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:30.804484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:30.805029 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:30.804912 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:31.414253 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:31.414216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:31.414405 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:31.414275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:31.414405 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.414372 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:31.414405 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.414381 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:31.414586 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.414452 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs podName:a541e958-955c-43a7-9572-845dfd6637ce nodeName:}" failed. No retries permitted until 2026-07-06 22:25:03.414437218 +0000 UTC m=+65.150839199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs") pod "network-metrics-daemon-dm8zz" (UID: "a541e958-955c-43a7-9572-845dfd6637ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 22:24:31.414586 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.414467 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret podName:9a70d449-b56c-4f5b-8af3-90734fa84198 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:03.414460765 +0000 UTC m=+65.150862729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret") pod "global-pull-secret-syncer-qb6jc" (UID: "9a70d449-b56c-4f5b-8af3-90734fa84198") : object "kube-system"/"original-pull-secret" not registered Jul 06 22:24:31.515349 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:31.515319 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:31.515529 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.515512 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 22:24:31.515586 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.515533 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 22:24:31.515586 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.515546 2564 projected.go:196] Error preparing data for projected volume kube-api-access-dg7h5 for pod openshift-network-diagnostics/network-check-target-j25q7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:31.515675 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:31.515609 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5 podName:8ec2a462-be6a-47b6-b96b-596143cbbb30 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:03.515587834 +0000 UTC m=+65.251989817 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dg7h5" (UniqueName: "kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5") pod "network-check-target-j25q7" (UID: "8ec2a462-be6a-47b6-b96b-596143cbbb30") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 22:24:32.422805 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.422767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:32.478927 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.478901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:32.804195 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.804117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:32.804345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.804125 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:32.804345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.804247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:32.804345 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:32.804247 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dm8zz" podUID="a541e958-955c-43a7-9572-845dfd6637ce" Jul 06 22:24:32.804525 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:32.804359 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-qb6jc" podUID="9a70d449-b56c-4f5b-8af3-90734fa84198" Jul 06 22:24:32.804525 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:32.804494 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-j25q7" podUID="8ec2a462-be6a-47b6-b96b-596143cbbb30" Jul 06 22:24:32.982711 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.982683 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-55.ec2.internal" event="NodeReady" Jul 06 22:24:32.982909 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:32.982799 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 22:24:33.023845 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.023821 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6ftjg"] Jul 06 22:24:33.026419 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.026396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.026813 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.026789 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lfhr9"] Jul 06 22:24:33.028877 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.028812 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 22:24:33.028877 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.028825 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.029027 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.028948 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 22:24:33.029027 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.028989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mrg4w\"" Jul 06 22:24:33.031288 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.031173 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 22:24:33.031288 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.031238 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 22:24:33.031587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.031471 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 22:24:33.031587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.031492 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zvrrt\"" Jul 06 22:24:33.036543 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.036524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6ftjg"] Jul 06 22:24:33.039114 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.039093 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfhr9"] Jul 06 22:24:33.126770 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.126730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwl7b\" (UniqueName: \"kubernetes.io/projected/1689f60b-7849-452d-ba1a-27a8057595a9-kube-api-access-bwl7b\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.126918 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.126791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1689f60b-7849-452d-ba1a-27a8057595a9-config-volume\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.126918 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.126821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.126918 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.126862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdhtc\" (UniqueName: \"kubernetes.io/projected/f21331f3-0785-4537-bcd4-9151b6de0dea-kube-api-access-tdhtc\") pod \"ingress-canary-lfhr9\" (UID: \"f21331f3-0785-4537-bcd4-9151b6de0dea\") " pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.127072 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.126988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f21331f3-0785-4537-bcd4-9151b6de0dea-cert\") pod \"ingress-canary-lfhr9\" (UID: \"f21331f3-0785-4537-bcd4-9151b6de0dea\") " pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.127123 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.127091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1689f60b-7849-452d-ba1a-27a8057595a9-tmp-dir\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.151479 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.151455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerStarted","Data":"954cd0ae785d79145992723f8657477110b1d01cbf88deda8616ac54db78cba3"} Jul 06 22:24:33.227403 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227374 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1689f60b-7849-452d-ba1a-27a8057595a9-tmp-dir\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.227535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwl7b\" (UniqueName: \"kubernetes.io/projected/1689f60b-7849-452d-ba1a-27a8057595a9-kube-api-access-bwl7b\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.227535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1689f60b-7849-452d-ba1a-27a8057595a9-config-volume\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.227535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.227535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdhtc\" (UniqueName: \"kubernetes.io/projected/f21331f3-0785-4537-bcd4-9151b6de0dea-kube-api-access-tdhtc\") pod \"ingress-canary-lfhr9\" (UID: \"f21331f3-0785-4537-bcd4-9151b6de0dea\") " pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.227736 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227587 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f21331f3-0785-4537-bcd4-9151b6de0dea-cert\") pod \"ingress-canary-lfhr9\" (UID: \"f21331f3-0785-4537-bcd4-9151b6de0dea\") " pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.227736 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:33.227635 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:33.227736 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:33.227713 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls podName:1689f60b-7849-452d-ba1a-27a8057595a9 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:33.727691445 +0000 UTC m=+35.464093430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls") pod "dns-default-6ftjg" (UID: "1689f60b-7849-452d-ba1a-27a8057595a9") : secret "dns-default-metrics-tls" not found Jul 06 22:24:33.227871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.227780 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1689f60b-7849-452d-ba1a-27a8057595a9-tmp-dir\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.228020 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.228000 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1689f60b-7849-452d-ba1a-27a8057595a9-config-volume\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.231517 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.231502 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f21331f3-0785-4537-bcd4-9151b6de0dea-cert\") pod \"ingress-canary-lfhr9\" (UID: \"f21331f3-0785-4537-bcd4-9151b6de0dea\") " pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.236517 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.236488 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwl7b\" (UniqueName: \"kubernetes.io/projected/1689f60b-7849-452d-ba1a-27a8057595a9-kube-api-access-bwl7b\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.236517 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.236503 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdhtc\" (UniqueName: \"kubernetes.io/projected/f21331f3-0785-4537-bcd4-9151b6de0dea-kube-api-access-tdhtc\") pod \"ingress-canary-lfhr9\" (UID: \"f21331f3-0785-4537-bcd4-9151b6de0dea\") " pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.352375 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.352352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lfhr9" Jul 06 22:24:33.432314 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.432286 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:33.476731 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.476569 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lfhr9"] Jul 06 22:24:33.480163 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:33.480139 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf21331f3_0785_4537_bcd4_9151b6de0dea.slice/crio-32634bd148e7d15878f0cb370a05fbd1ccf35836d84d2668567a0b9e023b21e8 WatchSource:0}: Error finding container 32634bd148e7d15878f0cb370a05fbd1ccf35836d84d2668567a0b9e023b21e8: Status 404 returned error can't find the container with id 32634bd148e7d15878f0cb370a05fbd1ccf35836d84d2668567a0b9e023b21e8 Jul 06 22:24:33.484608 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.484585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:33.730897 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:33.730808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:33.731028 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:33.730942 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:33.731028 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:33.730996 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls podName:1689f60b-7849-452d-ba1a-27a8057595a9 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:34.730980434 +0000 UTC m=+36.467382404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls") pod "dns-default-6ftjg" (UID: "1689f60b-7849-452d-ba1a-27a8057595a9") : secret "dns-default-metrics-tls" not found Jul 06 22:24:34.159236 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.155652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfhr9" event={"ID":"f21331f3-0785-4537-bcd4-9151b6de0dea","Type":"ContainerStarted","Data":"32634bd148e7d15878f0cb370a05fbd1ccf35836d84d2668567a0b9e023b21e8"} Jul 06 22:24:34.162025 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.161834 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c00558f-c19d-4d65-82c8-73dd2d6d094e" containerID="954cd0ae785d79145992723f8657477110b1d01cbf88deda8616ac54db78cba3" exitCode=0 Jul 06 22:24:34.162025 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.161893 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerDied","Data":"954cd0ae785d79145992723f8657477110b1d01cbf88deda8616ac54db78cba3"} Jul 06 22:24:34.175674 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.175651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:34.231613 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.231586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:34.740008 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.739970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:34.740453 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:34.740103 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:34.740453 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:34.740217 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls podName:1689f60b-7849-452d-ba1a-27a8057595a9 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:36.740199785 +0000 UTC m=+38.476601777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls") pod "dns-default-6ftjg" (UID: "1689f60b-7849-452d-ba1a-27a8057595a9") : secret "dns-default-metrics-tls" not found Jul 06 22:24:34.804075 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.803982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:24:34.804256 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.804123 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:24:34.804319 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.803982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:24:34.809505 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.809482 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:24:34.810864 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.810840 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 22:24:34.810994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.810852 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 22:24:34.812366 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.812352 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wdmk\"" Jul 06 22:24:34.813104 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.813089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czcdn\"" Jul 06 22:24:34.813494 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:34.813477 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:24:35.168147 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:35.168112 2564 generic.go:356] "Generic (PLEG): container finished" podID="3c00558f-c19d-4d65-82c8-73dd2d6d094e" containerID="d10277098c0ba2561a1832172f0e91f739bcbe211ba282f48bf89993a6dd6de6" exitCode=0 Jul 06 22:24:35.168325 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:35.168174 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerDied","Data":"d10277098c0ba2561a1832172f0e91f739bcbe211ba282f48bf89993a6dd6de6"} Jul 06 22:24:36.172341 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.172307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lfhr9" event={"ID":"f21331f3-0785-4537-bcd4-9151b6de0dea","Type":"ContainerStarted","Data":"0ebc6e1bac016ca8f6ba926fc9872f5776abb79f63224d7e0b42a40a9ae573ca"} Jul 06 22:24:36.175527 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.175502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" event={"ID":"3c00558f-c19d-4d65-82c8-73dd2d6d094e","Type":"ContainerStarted","Data":"9415649b69a12cdd9fa23acf81f21df2d1d6e6a4bc63a22dfbf691e92c557c16"} Jul 06 22:24:36.187792 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.187753 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lfhr9" podStartSLOduration=1.005096672 podStartE2EDuration="3.187739991s" totalImagesPullingTime="2.182643319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:36.186974635 +0000 UTC m=+37.923376615" watchObservedRunningTime="2026-07-06 22:24:36.187739991 +0000 UTC m=+37.924142011" Jul 06 22:24:36.208021 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.207976 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-zkjhk" podStartSLOduration=7.85477777 podStartE2EDuration="38.207957907s" totalImagesPullingTime="30.353180137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:36.206103093 +0000 UTC m=+37.942505081" watchObservedRunningTime="2026-07-06 22:24:36.207957907 +0000 UTC m=+37.944359898" Jul 06 22:24:36.326235 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.326201 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.426784 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.426592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:36.753705 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:36.753633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:36.753837 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:36.753770 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:36.753837 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:36.753835 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls podName:1689f60b-7849-452d-ba1a-27a8057595a9 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:40.753817707 +0000 UTC m=+42.490219672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls") pod "dns-default-6ftjg" (UID: "1689f60b-7849-452d-ba1a-27a8057595a9") : secret "dns-default-metrics-tls" not found Jul 06 22:24:38.233099 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:38.233057 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:38.288497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:38.288470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:40.781357 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:40.781323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:40.781779 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:40.781449 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:40.781779 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:40.781506 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls podName:1689f60b-7849-452d-ba1a-27a8057595a9 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:48.78149232 +0000 UTC m=+50.517894285 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls") pod "dns-default-6ftjg" (UID: "1689f60b-7849-452d-ba1a-27a8057595a9") : secret "dns-default-metrics-tls" not found Jul 06 22:24:41.490202 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.490160 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx"] Jul 06 22:24:41.512453 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.512409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx"] Jul 06 22:24:41.512595 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.512468 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh"] Jul 06 22:24:41.512640 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.512593 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" Jul 06 22:24:41.515349 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.515325 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:41.515495 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.515334 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 06 22:24:41.516876 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.516853 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-nzhp5\"" Jul 06 22:24:41.526462 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.526442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh"] Jul 06 22:24:41.526547 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.526501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:41.529008 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.528986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 06 22:24:41.529121 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.529011 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:41.529121 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.529106 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4gmlj\"" Jul 06 22:24:41.529210 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.529137 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 06 22:24:41.585260 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.585226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w6ff\" (UniqueName: \"kubernetes.io/projected/f267494d-23f5-4a1c-af81-b96eea8c0862-kube-api-access-8w6ff\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:41.585415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.585270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdlpj\" (UniqueName: \"kubernetes.io/projected/752a5490-71cf-4d8b-830a-2eb5c9560e5f-kube-api-access-qdlpj\") pod \"volume-data-source-validator-5d456fdbcd-wzcsx\" (UID: \"752a5490-71cf-4d8b-830a-2eb5c9560e5f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" Jul 06 22:24:41.585415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.585289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:41.613435 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.613404 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf"] Jul 06 22:24:41.636328 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.636301 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw"] Jul 06 22:24:41.636458 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.636433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.642543 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.642520 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-lwpmp\"" Jul 06 22:24:41.642761 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.642729 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 06 22:24:41.643579 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.643559 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 06 22:24:41.643812 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.643792 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 06 22:24:41.643812 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.643809 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 06 22:24:41.660466 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.660449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" Jul 06 22:24:41.660564 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.660451 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5fd494bcf9-kfm22"] Jul 06 22:24:41.662799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.662783 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bz492\"" Jul 06 22:24:41.678490 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.678472 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf"] Jul 06 22:24:41.678577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.678499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw"] Jul 06 22:24:41.678577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.678518 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66599dd4f7-vqp4q"] Jul 06 22:24:41.678642 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.678590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.683441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.683407 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 22:24:41.683599 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.683584 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 22:24:41.684348 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.684330 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 22:24:41.684451 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.684359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 22:24:41.684451 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.684415 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 22:24:41.684849 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.684835 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 22:24:41.685835 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.685818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.685921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.685847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhgt\" (UniqueName: \"kubernetes.io/projected/24907fdd-7f13-4aeb-bdde-365f931c8c82-kube-api-access-dbhgt\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.685921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.685876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:41.686001 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.685926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-config\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.686001 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.685972 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:24:41.686130 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686025 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.686130 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.686032 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls podName:f267494d-23f5-4a1c-af81-b96eea8c0862 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.18601524 +0000 UTC m=+43.922417207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tkpmh" (UID: "f267494d-23f5-4a1c-af81-b96eea8c0862") : secret "samples-operator-tls" not found Jul 06 22:24:41.686130 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.686130 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mjk\" (UniqueName: \"kubernetes.io/projected/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-kube-api-access-t5mjk\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.686331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686132 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-default-certificate\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.686331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24kt\" (UniqueName: \"kubernetes.io/projected/e4078953-8ace-489d-9810-ceb5b9cfc7a6-kube-api-access-d24kt\") pod \"network-check-source-989f8b8dc-s57dw\" (UID: \"e4078953-8ace-489d-9810-ceb5b9cfc7a6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" Jul 06 22:24:41.686331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686176 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w6ff\" (UniqueName: \"kubernetes.io/projected/f267494d-23f5-4a1c-af81-b96eea8c0862-kube-api-access-8w6ff\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:41.686331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-stats-auth\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.686331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686268 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdlpj\" (UniqueName: \"kubernetes.io/projected/752a5490-71cf-4d8b-830a-2eb5c9560e5f-kube-api-access-qdlpj\") pod \"volume-data-source-validator-5d456fdbcd-wzcsx\" (UID: \"752a5490-71cf-4d8b-830a-2eb5c9560e5f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" Jul 06 22:24:41.686331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.686314 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-sxzzb\"" Jul 06 22:24:41.692521 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.692506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fd494bcf9-kfm22"] Jul 06 22:24:41.692598 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.692589 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.695324 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.695298 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-62ptd\"" Jul 06 22:24:41.695474 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.695410 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 22:24:41.695474 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.695414 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 22:24:41.695474 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.695439 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 22:24:41.696218 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.696204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66599dd4f7-vqp4q"] Jul 06 22:24:41.703148 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.703128 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 22:24:41.706768 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.706749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w6ff\" (UniqueName: \"kubernetes.io/projected/f267494d-23f5-4a1c-af81-b96eea8c0862-kube-api-access-8w6ff\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:41.706848 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.706799 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdlpj\" (UniqueName: \"kubernetes.io/projected/752a5490-71cf-4d8b-830a-2eb5c9560e5f-kube-api-access-qdlpj\") pod \"volume-data-source-validator-5d456fdbcd-wzcsx\" (UID: \"752a5490-71cf-4d8b-830a-2eb5c9560e5f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" Jul 06 22:24:41.748089 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.748027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:41.787289 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-bound-sa-token\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787300 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mjk\" (UniqueName: \"kubernetes.io/projected/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-kube-api-access-t5mjk\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-image-registry-private-configuration\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-installation-pull-secrets\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787486 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppk8b\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-kube-api-access-ppk8b\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.787521 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.287499544 +0000 UTC m=+44.023901533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : configmap references non-existent config key: service-ca.crt Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-default-certificate\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787625 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24kt\" (UniqueName: \"kubernetes.io/projected/e4078953-8ace-489d-9810-ceb5b9cfc7a6-kube-api-access-d24kt\") pod \"network-check-source-989f8b8dc-s57dw\" (UID: \"e4078953-8ace-489d-9810-ceb5b9cfc7a6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-stats-auth\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.787707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhgt\" (UniqueName: \"kubernetes.io/projected/24907fdd-7f13-4aeb-bdde-365f931c8c82-kube-api-access-dbhgt\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.787909 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-config\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.787916 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.787978 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.287960721 +0000 UTC m=+44.024362693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : secret "router-metrics-certs-default" not found Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.788026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-trusted-ca\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.788080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a00d43cd-f3ab-41f1-b464-641c2fc88979-ca-trust-extracted\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.788296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.788111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-certificates\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.788680 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.788599 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-config\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.789695 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.789673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.790549 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.790528 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-stats-auth\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.790590 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.790533 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-default-certificate\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.798583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.798558 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24kt\" (UniqueName: \"kubernetes.io/projected/e4078953-8ace-489d-9810-ceb5b9cfc7a6-kube-api-access-d24kt\") pod \"network-check-source-989f8b8dc-s57dw\" (UID: \"e4078953-8ace-489d-9810-ceb5b9cfc7a6\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" Jul 06 22:24:41.800564 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.800545 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mjk\" (UniqueName: \"kubernetes.io/projected/1f4aca1d-2fa0-475b-bac8-b5903a9e68b1-kube-api-access-t5mjk\") pod \"service-ca-operator-5d46d9ffd4-k67cf\" (UID: \"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.800990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.800968 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhgt\" (UniqueName: \"kubernetes.io/projected/24907fdd-7f13-4aeb-bdde-365f931c8c82-kube-api-access-dbhgt\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:41.822942 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.822924 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" Jul 06 22:24:41.888609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-trusted-ca\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888722 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a00d43cd-f3ab-41f1-b464-641c2fc88979-ca-trust-extracted\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888722 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-certificates\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-bound-sa-token\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-image-registry-private-configuration\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-installation-pull-secrets\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888946 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppk8b\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-kube-api-access-ppk8b\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.888946 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.888919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.889035 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.889013 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a00d43cd-f3ab-41f1-b464-641c2fc88979-ca-trust-extracted\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.889085 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.889053 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:24:41.889085 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.889071 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66599dd4f7-vqp4q: secret "image-registry-tls" not found Jul 06 22:24:41.889174 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:41.889131 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls podName:a00d43cd-f3ab-41f1-b464-641c2fc88979 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:42.389113789 +0000 UTC m=+44.125515757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls") pod "image-registry-66599dd4f7-vqp4q" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979") : secret "image-registry-tls" not found Jul 06 22:24:41.889257 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.889236 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-certificates\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.889926 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.889904 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-trusted-ca\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.891124 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.891104 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-image-registry-private-configuration\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.891192 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.891176 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-installation-pull-secrets\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.897252 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.897228 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppk8b\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-kube-api-access-ppk8b\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.897637 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.897613 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-bound-sa-token\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:41.926028 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.926002 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:41.945771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.945736 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" Jul 06 22:24:41.968883 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.968863 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" Jul 06 22:24:41.980261 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.980238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx"] Jul 06 22:24:41.982276 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:41.982255 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752a5490_71cf_4d8b_830a_2eb5c9560e5f.slice/crio-ef9773f7a5b56a69deccd7e3881670b4e0ab195c8b62ee27954d0c03e6646fcc WatchSource:0}: Error finding container ef9773f7a5b56a69deccd7e3881670b4e0ab195c8b62ee27954d0c03e6646fcc: Status 404 returned error can't find the container with id ef9773f7a5b56a69deccd7e3881670b4e0ab195c8b62ee27954d0c03e6646fcc Jul 06 22:24:41.996655 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:41.996629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:42.087728 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.087618 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf"] Jul 06 22:24:42.091029 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:42.090997 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4aca1d_2fa0_475b_bac8_b5903a9e68b1.slice/crio-b3fe046168d81bf7df12e7be77770110e9fa7e16cf29566c12899fbe735def13 WatchSource:0}: Error finding container b3fe046168d81bf7df12e7be77770110e9fa7e16cf29566c12899fbe735def13: Status 404 returned error can't find the container with id b3fe046168d81bf7df12e7be77770110e9fa7e16cf29566c12899fbe735def13 Jul 06 22:24:42.131895 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.131870 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw"] Jul 06 22:24:42.137024 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:42.136991 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4078953_8ace_489d_9810_ceb5b9cfc7a6.slice/crio-7f36f3db7981f5a2e2493949c29597845cde0895bd61d68832237c6f3625d215 WatchSource:0}: Error finding container 7f36f3db7981f5a2e2493949c29597845cde0895bd61d68832237c6f3625d215: Status 404 returned error can't find the container with id 7f36f3db7981f5a2e2493949c29597845cde0895bd61d68832237c6f3625d215 Jul 06 22:24:42.190557 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.190527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" event={"ID":"e4078953-8ace-489d-9810-ceb5b9cfc7a6","Type":"ContainerStarted","Data":"7f36f3db7981f5a2e2493949c29597845cde0895bd61d68832237c6f3625d215"} Jul 06 22:24:42.191345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.191325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:42.191494 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.191481 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:24:42.191542 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.191533 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls podName:f267494d-23f5-4a1c-af81-b96eea8c0862 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:43.19151928 +0000 UTC m=+44.927921245 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tkpmh" (UID: "f267494d-23f5-4a1c-af81-b96eea8c0862") : secret "samples-operator-tls" not found Jul 06 22:24:42.191657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.191637 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" event={"ID":"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1","Type":"ContainerStarted","Data":"b3fe046168d81bf7df12e7be77770110e9fa7e16cf29566c12899fbe735def13"} Jul 06 22:24:42.192636 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.192617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" event={"ID":"752a5490-71cf-4d8b-830a-2eb5c9560e5f","Type":"ContainerStarted","Data":"ef9773f7a5b56a69deccd7e3881670b4e0ab195c8b62ee27954d0c03e6646fcc"} Jul 06 22:24:42.292735 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.292654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:42.292735 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.292718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:42.292936 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.292818 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 22:24:42.292936 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.292824 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:43.292811839 +0000 UTC m=+45.029213804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : configmap references non-existent config key: service-ca.crt Jul 06 22:24:42.292936 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.292890 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:43.29287275 +0000 UTC m=+45.029274720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : secret "router-metrics-certs-default" not found Jul 06 22:24:42.394009 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:42.393978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:42.394162 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.394124 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:24:42.394162 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.394143 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66599dd4f7-vqp4q: secret "image-registry-tls" not found Jul 06 22:24:42.394235 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:42.394193 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls podName:a00d43cd-f3ab-41f1-b464-641c2fc88979 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:43.394177878 +0000 UTC m=+45.130579844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls") pod "image-registry-66599dd4f7-vqp4q" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979") : secret "image-registry-tls" not found Jul 06 22:24:43.201441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:43.201395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:43.201985 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.201531 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:24:43.201985 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.201611 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls podName:f267494d-23f5-4a1c-af81-b96eea8c0862 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:45.201587617 +0000 UTC m=+46.937989591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tkpmh" (UID: "f267494d-23f5-4a1c-af81-b96eea8c0862") : secret "samples-operator-tls" not found Jul 06 22:24:43.302627 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:43.302588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:43.302789 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:43.302712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:43.302789 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.302724 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:45.302706122 +0000 UTC m=+47.039108091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : configmap references non-existent config key: service-ca.crt Jul 06 22:24:43.302909 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.302808 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 22:24:43.302909 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.302864 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:45.302849484 +0000 UTC m=+47.039251452 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : secret "router-metrics-certs-default" not found Jul 06 22:24:43.403748 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:43.403720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:43.403989 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.403972 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:24:43.404052 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.403992 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66599dd4f7-vqp4q: secret "image-registry-tls" not found Jul 06 22:24:43.404092 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:43.404058 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls podName:a00d43cd-f3ab-41f1-b464-641c2fc88979 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:45.40403916 +0000 UTC m=+47.140441132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls") pod "image-registry-66599dd4f7-vqp4q" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979") : secret "image-registry-tls" not found Jul 06 22:24:44.200751 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:44.200710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" event={"ID":"752a5490-71cf-4d8b-830a-2eb5c9560e5f","Type":"ContainerStarted","Data":"e2056c4ae25c74cbad2892d7983dd18d873005d6f8e4749f1ff8329444d11b5e"} Jul 06 22:24:44.217356 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:44.217303 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-wzcsx" podStartSLOduration=1.697490886 podStartE2EDuration="3.217287289s" totalImagesPullingTime="1.519796403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:44.215247614 +0000 UTC m=+45.951649602" watchObservedRunningTime="2026-07-06 22:24:44.217287289 +0000 UTC m=+45.953689276" Jul 06 22:24:44.319082 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:44.319053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:44.419744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:44.419714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:45.218440 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:45.218392 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:45.218901 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.218586 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:24:45.218901 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.218655 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls podName:f267494d-23f5-4a1c-af81-b96eea8c0862 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:49.218638243 +0000 UTC m=+50.955040228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tkpmh" (UID: "f267494d-23f5-4a1c-af81-b96eea8c0862") : secret "samples-operator-tls" not found Jul 06 22:24:45.319260 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:45.319230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:45.319466 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:45.319412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:45.319466 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.319445 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:49.319407223 +0000 UTC m=+51.055809193 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : configmap references non-existent config key: service-ca.crt Jul 06 22:24:45.319600 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.319516 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 22:24:45.319600 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.319589 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:49.319547066 +0000 UTC m=+51.055949033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : secret "router-metrics-certs-default" not found Jul 06 22:24:45.420370 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:45.420336 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:45.420557 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.420488 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:24:45.420557 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.420503 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66599dd4f7-vqp4q: secret "image-registry-tls" not found Jul 06 22:24:45.420557 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:45.420551 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls podName:a00d43cd-f3ab-41f1-b464-641c2fc88979 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:49.420537694 +0000 UTC m=+51.156939659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls") pod "image-registry-66599dd4f7-vqp4q" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979") : secret "image-registry-tls" not found Jul 06 22:24:46.207766 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:46.207725 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" event={"ID":"e4078953-8ace-489d-9810-ceb5b9cfc7a6","Type":"ContainerStarted","Data":"7a5b6db404408a80197d5d3cfc5130f8559d0076c40d01ef8523f502d12bef74"} Jul 06 22:24:46.209381 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:46.209355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" event={"ID":"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1","Type":"ContainerStarted","Data":"1d4a05076cca7fe10a4303db8e36078c68116d42a8aa1de32f60c7e93a87df5f"} Jul 06 22:24:46.221675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:46.221626 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-s57dw" podStartSLOduration=1.278982331 podStartE2EDuration="5.221609798s" totalImagesPullingTime="3.942627467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:46.221332255 +0000 UTC m=+47.957734243" watchObservedRunningTime="2026-07-06 22:24:46.221609798 +0000 UTC m=+47.958011786" Jul 06 22:24:46.237016 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:46.236978 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" podStartSLOduration=1.5960763569999998 podStartE2EDuration="5.236964922s" totalImagesPullingTime="3.640888565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:46.236066234 +0000 UTC m=+47.972468223" watchObservedRunningTime="2026-07-06 22:24:46.236964922 +0000 UTC m=+47.973366908" Jul 06 22:24:46.812289 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:46.812254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:46.914595 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:46.914562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:47.054316 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.054287 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bhpss"] Jul 06 22:24:47.069286 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.069224 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhpss"] Jul 06 22:24:47.069406 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.069318 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.072465 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.072444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 22:24:47.072694 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.072677 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 22:24:47.074009 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.073989 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 22:24:47.074009 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.074005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bgz5b\"" Jul 06 22:24:47.074160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.074033 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 22:24:47.077386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.077363 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:47.132556 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.132531 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-data-volume\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.132704 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.132638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.132752 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.132712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-crio-socket\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.132752 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.132738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfc8c\" (UniqueName: \"kubernetes.io/projected/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-api-access-rfc8c\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.132880 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.132764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.141413 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.141378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:47.233890 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.233859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-crio-socket\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.233890 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.233890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfc8c\" (UniqueName: \"kubernetes.io/projected/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-api-access-rfc8c\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.234345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.233915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.234345 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.233971 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:47.234345 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.234025 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:47.734009515 +0000 UTC m=+49.470411480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : configmap "kube-rbac-proxy" not found Jul 06 22:24:47.234345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.234062 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-crio-socket\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.234345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.234157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-data-volume\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.234345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.234285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.234587 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.234391 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:47.234587 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.234463 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:47.734446514 +0000 UTC m=+49.470848491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:47.234587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.234540 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-data-volume\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.242615 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.242589 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfc8c\" (UniqueName: \"kubernetes.io/projected/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-api-access-rfc8c\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.491347 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.491301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:47.738772 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.738734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.738935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.738901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:47.738935 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.738911 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:47.739053 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.738977 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:47.739053 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.738985 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:48.738962417 +0000 UTC m=+50.475364405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:47.739053 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:47.739015 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:48.739004398 +0000 UTC m=+50.475406363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : configmap "kube-rbac-proxy" not found Jul 06 22:24:47.851493 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.851461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:47.931800 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:47.931768 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:48.747858 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:48.747814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:48.748254 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:48.747890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:48.748254 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:48.747980 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:48.748254 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:48.748025 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:48.748254 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:48.748047 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:50.748030925 +0000 UTC m=+52.484432905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:48.748254 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:48.748080 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:50.748066594 +0000 UTC m=+52.484468560 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : configmap "kube-rbac-proxy" not found Jul 06 22:24:48.836301 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:48.836272 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nsz97_caaf6673-9f01-40c6-a0ee-fe4992b9385a/dns-node-resolver/0.log" Jul 06 22:24:48.848650 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:48.848619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:24:48.848802 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:48.848787 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 22:24:48.848857 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:48.848846 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls podName:1689f60b-7849-452d-ba1a-27a8057595a9 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:04.848830666 +0000 UTC m=+66.585232648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls") pod "dns-default-6ftjg" (UID: "1689f60b-7849-452d-ba1a-27a8057595a9") : secret "dns-default-metrics-tls" not found Jul 06 22:24:49.252141 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.252091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:49.252344 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.252246 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 06 22:24:49.252344 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.252313 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls podName:f267494d-23f5-4a1c-af81-b96eea8c0862 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:57.252296792 +0000 UTC m=+58.988698758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tkpmh" (UID: "f267494d-23f5-4a1c-af81-b96eea8c0862") : secret "samples-operator-tls" not found Jul 06 22:24:49.353160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.353117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:49.353386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.353217 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:49.353386 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.353310 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 22:24:49.353386 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.353306 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:57.353282619 +0000 UTC m=+59.089684594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : configmap references non-existent config key: service-ca.crt Jul 06 22:24:49.353599 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.353502 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs podName:24907fdd-7f13-4aeb-bdde-365f931c8c82 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:57.353349961 +0000 UTC m=+59.089751926 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs") pod "router-default-5fd494bcf9-kfm22" (UID: "24907fdd-7f13-4aeb-bdde-365f931c8c82") : secret "router-metrics-certs-default" not found Jul 06 22:24:49.363833 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.363806 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-j4c4z"] Jul 06 22:24:49.373463 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.373417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.377125 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.377098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-j4c4z"] Jul 06 22:24:49.377415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.377313 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 06 22:24:49.377415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.377378 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 06 22:24:49.377415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.377389 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 06 22:24:49.377621 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.377389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5nfc6\"" Jul 06 22:24:49.378108 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.378095 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 06 22:24:49.407308 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.407279 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:49.454030 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.453989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:49.454220 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.454053 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9e107d4d-55b0-4eaa-b170-46ac07ecea49-signing-key\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.454220 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.454094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9e107d4d-55b0-4eaa-b170-46ac07ecea49-signing-cabundle\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.454220 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.454137 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 22:24:49.454220 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.454152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8cc\" (UniqueName: \"kubernetes.io/projected/9e107d4d-55b0-4eaa-b170-46ac07ecea49-kube-api-access-hm8cc\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.454220 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.454159 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66599dd4f7-vqp4q: secret "image-registry-tls" not found Jul 06 22:24:49.454381 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:49.454229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls podName:a00d43cd-f3ab-41f1-b464-641c2fc88979 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:57.454212721 +0000 UTC m=+59.190614687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls") pod "image-registry-66599dd4f7-vqp4q" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979") : secret "image-registry-tls" not found Jul 06 22:24:49.508235 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.508162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:49.554776 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.554730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9e107d4d-55b0-4eaa-b170-46ac07ecea49-signing-cabundle\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.554948 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.554794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8cc\" (UniqueName: \"kubernetes.io/projected/9e107d4d-55b0-4eaa-b170-46ac07ecea49-kube-api-access-hm8cc\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.554948 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.554892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9e107d4d-55b0-4eaa-b170-46ac07ecea49-signing-key\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.555385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.555365 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9e107d4d-55b0-4eaa-b170-46ac07ecea49-signing-cabundle\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.557199 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.557172 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9e107d4d-55b0-4eaa-b170-46ac07ecea49-signing-key\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.563658 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.563628 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8cc\" (UniqueName: \"kubernetes.io/projected/9e107d4d-55b0-4eaa-b170-46ac07ecea49-kube-api-access-hm8cc\") pod \"service-ca-758f599c5-j4c4z\" (UID: \"9e107d4d-55b0-4eaa-b170-46ac07ecea49\") " pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.637272 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.637243 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g8jwm_b0aa773f-d298-4bde-a4fb-95518c2339ac/node-ca/0.log" Jul 06 22:24:49.683565 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.683525 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-j4c4z" Jul 06 22:24:49.826249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:49.826219 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-j4c4z"] Jul 06 22:24:49.829061 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:49.829032 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e107d4d_55b0_4eaa_b170_46ac07ecea49.slice/crio-d17c8f4b5afa9b57d060f4b2880fd3bb10578ccf5d6815efd8c9a5004c55a67c WatchSource:0}: Error finding container d17c8f4b5afa9b57d060f4b2880fd3bb10578ccf5d6815efd8c9a5004c55a67c: Status 404 returned error can't find the container with id d17c8f4b5afa9b57d060f4b2880fd3bb10578ccf5d6815efd8c9a5004c55a67c Jul 06 22:24:50.220966 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.220930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-j4c4z" event={"ID":"9e107d4d-55b0-4eaa-b170-46ac07ecea49","Type":"ContainerStarted","Data":"f4561f892f41a463629b2372d464180a9d2f96b7b47d7051fbd8742ed9d83c55"} Jul 06 22:24:50.221126 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.220979 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-j4c4z" event={"ID":"9e107d4d-55b0-4eaa-b170-46ac07ecea49","Type":"ContainerStarted","Data":"d17c8f4b5afa9b57d060f4b2880fd3bb10578ccf5d6815efd8c9a5004c55a67c"} Jul 06 22:24:50.236842 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.236818 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfhr9_f21331f3-0785-4537-bcd4-9151b6de0dea/serve-healthcheck-canary/0.log" Jul 06 22:24:50.243886 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.243843 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-j4c4z" podStartSLOduration=1.243829239 podStartE2EDuration="1.243829239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:50.242120614 +0000 UTC m=+51.978522626" watchObservedRunningTime="2026-07-06 22:24:50.243829239 +0000 UTC m=+51.980231225" Jul 06 22:24:50.332610 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.332581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:50.766180 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.766136 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:50.766355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:50.766212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:50.766355 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:50.766309 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:50.766355 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:50.766327 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:50.766580 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:50.766394 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:54.766372571 +0000 UTC m=+56.502774537 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:50.766580 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:50.766415 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:24:54.766406429 +0000 UTC m=+56.502808400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : configmap "kube-rbac-proxy" not found Jul 06 22:24:51.184473 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:51.184402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:51.308940 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:51.308899 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:51.410484 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:51.410449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:51.532070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:51.531973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:51.705401 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:51.705368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:52.071416 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:52.071378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:52.666679 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:52.666647 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:53.071896 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:53.071815 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:53.341107 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:53.341077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:53.466611 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:53.466579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:53.673750 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:53.673662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:54.268009 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:54.267975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:54.679063 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:54.679032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:54.799795 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:54.799756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:54.799954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:54.799813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:24:54.799954 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:54.799902 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 22:24:54.799954 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:54.799951 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:02.799937965 +0000 UTC m=+64.536339933 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : configmap "kube-rbac-proxy" not found Jul 06 22:24:54.800095 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:54.799903 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 22:24:54.800095 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:24:54.799982 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls podName:8382f3d5-bbd9-4ce9-9b35-81a3debfa308 nodeName:}" failed. No retries permitted until 2026-07-06 22:25:02.79997604 +0000 UTC m=+64.536378005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bhpss" (UID: "8382f3d5-bbd9-4ce9-9b35-81a3debfa308") : secret "insights-runtime-extractor-tls" not found Jul 06 22:24:54.975286 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:54.975199 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.078443 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:55.078401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.178274 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:55.178237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.477698 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:55.477665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.674594 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:55.674561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:55.774935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:55.774848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.097633 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:56.097605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:56.669415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:56.669381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:57.276960 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.276920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:57.321614 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.321581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:57.323881 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.323852 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f267494d-23f5-4a1c-af81-b96eea8c0862-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tkpmh\" (UID: \"f267494d-23f5-4a1c-af81-b96eea8c0862\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:57.422539 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.422505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:57.422716 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.422620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:57.423262 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.423235 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24907fdd-7f13-4aeb-bdde-365f931c8c82-service-ca-bundle\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:57.425114 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.425090 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24907fdd-7f13-4aeb-bdde-365f931c8c82-metrics-certs\") pod \"router-default-5fd494bcf9-kfm22\" (UID: \"24907fdd-7f13-4aeb-bdde-365f931c8c82\") " pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:57.434636 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.434616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" Jul 06 22:24:57.473508 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.473483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:57.524094 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.524059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:57.527107 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.527009 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"image-registry-66599dd4f7-vqp4q\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:57.559979 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.559939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:57.586194 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.586175 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:57.596396 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.596373 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh"] Jul 06 22:24:57.602312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.602293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:57.748182 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.748153 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5fd494bcf9-kfm22"] Jul 06 22:24:57.751399 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:57.751371 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24907fdd_7f13_4aeb_bdde_365f931c8c82.slice/crio-3f4723d864918700a80a904768a13be4f55ffea496c643fa8d3bdbff23d0b27f WatchSource:0}: Error finding container 3f4723d864918700a80a904768a13be4f55ffea496c643fa8d3bdbff23d0b27f: Status 404 returned error can't find the container with id 3f4723d864918700a80a904768a13be4f55ffea496c643fa8d3bdbff23d0b27f Jul 06 22:24:57.766675 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.766651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66599dd4f7-vqp4q"] Jul 06 22:24:57.769242 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:24:57.769221 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00d43cd_f3ab_41f1_b464_641c2fc88979.slice/crio-8f89661b064dd4f4bd1cf20fff03bcf42712226fc6c2004a23cfca57944804e0 WatchSource:0}: Error finding container 8f89661b064dd4f4bd1cf20fff03bcf42712226fc6c2004a23cfca57944804e0: Status 404 returned error can't find the container with id 8f89661b064dd4f4bd1cf20fff03bcf42712226fc6c2004a23cfca57944804e0 Jul 06 22:24:57.877731 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:57.877699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:58.073496 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.073406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:58.249482 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.249449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" event={"ID":"f267494d-23f5-4a1c-af81-b96eea8c0862","Type":"ContainerStarted","Data":"a9f853ff8d227dbd795ac1951aa9ad20f0982790e2e2caaf2d896ead21292533"} Jul 06 22:24:58.251254 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.251218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" event={"ID":"a00d43cd-f3ab-41f1-b464-641c2fc88979","Type":"ContainerStarted","Data":"24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8"} Jul 06 22:24:58.251390 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.251280 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" event={"ID":"a00d43cd-f3ab-41f1-b464-641c2fc88979","Type":"ContainerStarted","Data":"8f89661b064dd4f4bd1cf20fff03bcf42712226fc6c2004a23cfca57944804e0"} Jul 06 22:24:58.251496 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.251415 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:24:58.252838 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.252814 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" event={"ID":"24907fdd-7f13-4aeb-bdde-365f931c8c82","Type":"ContainerStarted","Data":"73a89c95bd382596715d700a084efef27edf8d0b4f40297e0d84912f66b3795a"} Jul 06 22:24:58.252932 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.252850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" event={"ID":"24907fdd-7f13-4aeb-bdde-365f931c8c82","Type":"ContainerStarted","Data":"3f4723d864918700a80a904768a13be4f55ffea496c643fa8d3bdbff23d0b27f"} Jul 06 22:24:58.270752 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.270678 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" podStartSLOduration=17.270657369 podStartE2EDuration="17.270657369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:58.269485433 +0000 UTC m=+60.005887421" watchObservedRunningTime="2026-07-06 22:24:58.270657369 +0000 UTC m=+60.007059358" Jul 06 22:24:58.289356 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.289293 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" podStartSLOduration=17.289275492 podStartE2EDuration="17.289275492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:24:58.287843357 +0000 UTC m=+60.024245345" watchObservedRunningTime="2026-07-06 22:24:58.289275492 +0000 UTC m=+60.025677478" Jul 06 22:24:58.586954 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.586922 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:58.589332 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.589312 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:58.666359 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:58.666306 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:59.151497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:59.151467 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4fhw9" Jul 06 22:24:59.259152 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:59.259117 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:59.260458 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:59.260419 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5fd494bcf9-kfm22" Jul 06 22:24:59.276339 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:59.276309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:59.470763 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:59.470683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:24:59.917852 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:24:59.917819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:00.065051 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:00.064970 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:00.189348 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:00.189310 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:00.264113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:00.264079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" event={"ID":"f267494d-23f5-4a1c-af81-b96eea8c0862","Type":"ContainerStarted","Data":"7e09fd8214ed06eb6795e4fd8c79e2fada54075667fd938a3878783f9f554da4"} Jul 06 22:25:00.264324 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:00.264141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" event={"ID":"f267494d-23f5-4a1c-af81-b96eea8c0862","Type":"ContainerStarted","Data":"f65cf0cd17b98b7db437f326dc2852fd6fb75c143b257ece8efec9cff1d9cd4e"} Jul 06 22:25:00.282126 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:00.282077 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tkpmh" podStartSLOduration=17.286778707 podStartE2EDuration="19.282063307s" totalImagesPullingTime="1.9952846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:00.281527342 +0000 UTC m=+62.017929329" watchObservedRunningTime="2026-07-06 22:25:00.282063307 +0000 UTC m=+62.018465293" Jul 06 22:25:00.708310 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:00.708275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:01.074411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:01.074293 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:01.515113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:01.515079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:01.675149 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:01.675115 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:01.919635 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:01.919603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:01.975810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:01.975767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:02.871441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:02.871388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:25:02.871626 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:02.871463 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:25:02.871938 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:02.871895 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:25:02.873696 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:02.873673 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8382f3d5-bbd9-4ce9-9b35-81a3debfa308-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhpss\" (UID: \"8382f3d5-bbd9-4ce9-9b35-81a3debfa308\") " pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:25:02.981133 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:02.981098 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bgz5b\"" Jul 06 22:25:02.989271 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:02.989251 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhpss" Jul 06 22:25:03.113007 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.112965 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhpss"] Jul 06 22:25:03.117287 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:03.117257 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8382f3d5_bbd9_4ce9_9b35_81a3debfa308.slice/crio-fda556ace30d146b1e25dd02e44d8cae5803ac25bcdec5001c90e7f61b8045d2 WatchSource:0}: Error finding container fda556ace30d146b1e25dd02e44d8cae5803ac25bcdec5001c90e7f61b8045d2: Status 404 returned error can't find the container with id fda556ace30d146b1e25dd02e44d8cae5803ac25bcdec5001c90e7f61b8045d2 Jul 06 22:25:03.279384 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.279293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhpss" event={"ID":"8382f3d5-bbd9-4ce9-9b35-81a3debfa308","Type":"ContainerStarted","Data":"fda556ace30d146b1e25dd02e44d8cae5803ac25bcdec5001c90e7f61b8045d2"} Jul 06 22:25:03.477940 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.477904 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:25:03.478132 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.477956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:25:03.480512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.480493 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 22:25:03.481463 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.481449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 22:25:03.490752 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.490729 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a541e958-955c-43a7-9572-845dfd6637ce-metrics-certs\") pod \"network-metrics-daemon-dm8zz\" (UID: \"a541e958-955c-43a7-9572-845dfd6637ce\") " pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:25:03.490848 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.490822 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a70d449-b56c-4f5b-8af3-90734fa84198-original-pull-secret\") pod \"global-pull-secret-syncer-qb6jc\" (UID: \"9a70d449-b56c-4f5b-8af3-90734fa84198\") " pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:25:03.578692 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.578656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:25:03.582229 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.582152 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg7h5\" (UniqueName: \"kubernetes.io/projected/8ec2a462-be6a-47b6-b96b-596143cbbb30-kube-api-access-dg7h5\") pod \"network-check-target-j25q7\" (UID: \"8ec2a462-be6a-47b6-b96b-596143cbbb30\") " pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:25:03.616595 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.616558 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7wdmk\"" Jul 06 22:25:03.622163 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.621921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-qb6jc" Jul 06 22:25:03.624973 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.624951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dm8zz" Jul 06 22:25:03.630514 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.630488 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czcdn\"" Jul 06 22:25:03.638184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.638151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:25:03.830500 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.830266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-qb6jc"] Jul 06 22:25:03.837038 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:03.837008 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70d449_b56c_4f5b_8af3_90734fa84198.slice/crio-94ebe17107b51bf739ffd754b7bdf5a9b3a2f000c4c182ca177c099fe57d0de7 WatchSource:0}: Error finding container 94ebe17107b51bf739ffd754b7bdf5a9b3a2f000c4c182ca177c099fe57d0de7: Status 404 returned error can't find the container with id 94ebe17107b51bf739ffd754b7bdf5a9b3a2f000c4c182ca177c099fe57d0de7 Jul 06 22:25:03.860074 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:03.860040 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-j25q7"] Jul 06 22:25:03.864919 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:03.864886 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec2a462_be6a_47b6_b96b_596143cbbb30.slice/crio-4ca9e8b4d8e3934b40d11e06d3d1f6cb3947f279df85e30654dfbb5bbe4d36d2 WatchSource:0}: Error finding container 4ca9e8b4d8e3934b40d11e06d3d1f6cb3947f279df85e30654dfbb5bbe4d36d2: Status 404 returned error can't find the container with id 4ca9e8b4d8e3934b40d11e06d3d1f6cb3947f279df85e30654dfbb5bbe4d36d2 Jul 06 22:25:04.065250 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.065134 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dm8zz"] Jul 06 22:25:04.072543 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:04.072511 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda541e958_955c_43a7_9572_845dfd6637ce.slice/crio-24bf9c705f6e3ad8f39b875828c43c559368afb8219267976cc1142df6b71cc1 WatchSource:0}: Error finding container 24bf9c705f6e3ad8f39b875828c43c559368afb8219267976cc1142df6b71cc1: Status 404 returned error can't find the container with id 24bf9c705f6e3ad8f39b875828c43c559368afb8219267976cc1142df6b71cc1 Jul 06 22:25:04.184502 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.184377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:04.242065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.242032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:04.286479 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.286419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j25q7" event={"ID":"8ec2a462-be6a-47b6-b96b-596143cbbb30","Type":"ContainerStarted","Data":"8648308e82652758569ac1f68f08a85cbdbed1d368b24e9d0eee92c3343ca9af"} Jul 06 22:25:04.286664 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.286504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-j25q7" event={"ID":"8ec2a462-be6a-47b6-b96b-596143cbbb30","Type":"ContainerStarted","Data":"4ca9e8b4d8e3934b40d11e06d3d1f6cb3947f279df85e30654dfbb5bbe4d36d2"} Jul 06 22:25:04.287213 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.287186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:25:04.289022 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.288997 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dm8zz" event={"ID":"a541e958-955c-43a7-9572-845dfd6637ce","Type":"ContainerStarted","Data":"24bf9c705f6e3ad8f39b875828c43c559368afb8219267976cc1142df6b71cc1"} Jul 06 22:25:04.290200 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.290169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qb6jc" event={"ID":"9a70d449-b56c-4f5b-8af3-90734fa84198","Type":"ContainerStarted","Data":"94ebe17107b51bf739ffd754b7bdf5a9b3a2f000c4c182ca177c099fe57d0de7"} Jul 06 22:25:04.303062 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.303014 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-j25q7" podStartSLOduration=66.302996636 podStartE2EDuration="1m6.302996636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:04.301545654 +0000 UTC m=+66.037947645" watchObservedRunningTime="2026-07-06 22:25:04.302996636 +0000 UTC m=+66.039398624" Jul 06 22:25:04.896180 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.896052 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:25:04.900178 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:04.900123 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1689f60b-7849-452d-ba1a-27a8057595a9-metrics-tls\") pod \"dns-default-6ftjg\" (UID: \"1689f60b-7849-452d-ba1a-27a8057595a9\") " pod="openshift-dns/dns-default-6ftjg" Jul 06 22:25:05.143327 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:05.143289 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mrg4w\"" Jul 06 22:25:05.150556 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:05.150488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ftjg" Jul 06 22:25:06.183782 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:06.183746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:06.285141 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:06.285103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:06.385613 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:06.385579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:06.732050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:06.732021 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:06.858758 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:06.858720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:06.917249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:06.917218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.067568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:07.067483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.671698 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:07.671663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.776494 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:07.776461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:07.877341 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:07.877308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:08.161744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:08.161715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:08.257713 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:08.257680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:08.354756 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:08.354722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:09.215378 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.215342 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6ftjg"] Jul 06 22:25:09.220109 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:09.220064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1689f60b_7849_452d_ba1a_27a8057595a9.slice/crio-a5406cb9581392bccac556b4b8699cd55f59670ffdc61a544c1763bd01e17e53 WatchSource:0}: Error finding container a5406cb9581392bccac556b4b8699cd55f59670ffdc61a544c1763bd01e17e53: Status 404 returned error can't find the container with id a5406cb9581392bccac556b4b8699cd55f59670ffdc61a544c1763bd01e17e53 Jul 06 22:25:09.311557 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.311519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ftjg" event={"ID":"1689f60b-7849-452d-ba1a-27a8057595a9","Type":"ContainerStarted","Data":"a5406cb9581392bccac556b4b8699cd55f59670ffdc61a544c1763bd01e17e53"} Jul 06 22:25:09.315230 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.315194 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dm8zz" event={"ID":"a541e958-955c-43a7-9572-845dfd6637ce","Type":"ContainerStarted","Data":"6b779c8532895304a0c20302d843696b55d1b386ee59dfe6d9ed04c0523fb974"} Jul 06 22:25:09.315363 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.315250 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dm8zz" event={"ID":"a541e958-955c-43a7-9572-845dfd6637ce","Type":"ContainerStarted","Data":"9316f78f6b0786004f8bf1edd4c98915d03e7ed5e9abbf212a6c1de1377a9c4c"} Jul 06 22:25:09.317290 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.317235 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-qb6jc" event={"ID":"9a70d449-b56c-4f5b-8af3-90734fa84198","Type":"ContainerStarted","Data":"7e318a20565ded4c16f912bcda93000fab4bf2c7ad624ade45fb338f4c501b38"} Jul 06 22:25:09.318823 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.318779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhpss" event={"ID":"8382f3d5-bbd9-4ce9-9b35-81a3debfa308","Type":"ContainerStarted","Data":"31bf63e2242bd25b5fe7e3f1c30b1279f5458f4d92ec5cff19d11a6f425680eb"} Jul 06 22:25:09.331062 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.331010 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dm8zz" podStartSLOduration=66.336859114 podStartE2EDuration="1m11.330995953s" totalImagesPullingTime="4.994136839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:09.330932503 +0000 UTC m=+71.067334490" watchObservedRunningTime="2026-07-06 22:25:09.330995953 +0000 UTC m=+71.067397940" Jul 06 22:25:09.345584 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.345540 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-qb6jc" podStartSLOduration=66.096125593 podStartE2EDuration="1m11.345526703s" totalImagesPullingTime="5.24940111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:09.344898644 +0000 UTC m=+71.081300633" watchObservedRunningTime="2026-07-06 22:25:09.345526703 +0000 UTC m=+71.081928689" Jul 06 22:25:09.606398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.606350 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c"] Jul 06 22:25:09.608505 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.608482 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x"] Jul 06 22:25:09.608655 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.608629 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.610666 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.610645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.611860 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.611841 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 22:25:09.614641 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.614463 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 22:25:09.614761 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.614738 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 22:25:09.614761 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.614747 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 22:25:09.614876 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.614784 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 22:25:09.615070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.615038 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 06 22:25:09.615070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.615058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 22:25:09.615765 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.615620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 22:25:09.623865 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.623840 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66599dd4f7-vqp4q"] Jul 06 22:25:09.629574 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.629259 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c"] Jul 06 22:25:09.630366 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.630335 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x"] Jul 06 22:25:09.732219 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.732183 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7774bcc-qfphc"] Jul 06 22:25:09.735808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.735771 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.738907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw2qp\" (UniqueName: \"kubernetes.io/projected/f6995542-c356-4167-b9ab-0cf2c75d1c74-kube-api-access-zw2qp\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739166 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-hub\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f6995542-c356-4167-b9ab-0cf2c75d1c74-klusterlet-config\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739249 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb4q8\" (UniqueName: \"kubernetes.io/projected/0d2afc94-4048-4a25-b388-db8926951a27-kube-api-access-zb4q8\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739288 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2afc94-4048-4a25-b388-db8926951a27-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.739312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.739787 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-ca\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.739787 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.739391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6995542-c356-4167-b9ab-0cf2c75d1c74-tmp\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.745135 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.745104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:09.753149 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.753123 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7774bcc-qfphc"] Jul 06 22:25:09.822660 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.822617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:09.840033 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.839996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-hub\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.840227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840048 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f330c5c0-7c8e-47b4-8924-7414a02f502e-ca-trust-extracted\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.840227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840082 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f6995542-c356-4167-b9ab-0cf2c75d1c74-klusterlet-config\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.840227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840114 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f330c5c0-7c8e-47b4-8924-7414a02f502e-registry-certificates\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.840227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb4q8\" (UniqueName: \"kubernetes.io/projected/0d2afc94-4048-4a25-b388-db8926951a27-kube-api-access-zb4q8\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.840227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrgx\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-kube-api-access-kdrgx\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.840227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2afc94-4048-4a25-b388-db8926951a27-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840248 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-ca\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6995542-c356-4167-b9ab-0cf2c75d1c74-tmp\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-bound-sa-token\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840409 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw2qp\" (UniqueName: \"kubernetes.io/projected/f6995542-c356-4167-b9ab-0cf2c75d1c74-kube-api-access-zw2qp\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840526 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-registry-tls\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f330c5c0-7c8e-47b4-8924-7414a02f502e-image-registry-private-configuration\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.840587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f330c5c0-7c8e-47b4-8924-7414a02f502e-installation-pull-secrets\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.841018 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.840610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f330c5c0-7c8e-47b4-8924-7414a02f502e-trusted-ca\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.841095 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.841041 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d2afc94-4048-4a25-b388-db8926951a27-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.841986 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.841676 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6995542-c356-4167-b9ab-0cf2c75d1c74-tmp\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.843758 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.843545 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-hub\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.844372 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.844160 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-ca\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.844372 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.844317 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f6995542-c356-4167-b9ab-0cf2c75d1c74-klusterlet-config\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.844697 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.844653 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.845826 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.845802 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0d2afc94-4048-4a25-b388-db8926951a27-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.849315 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.849286 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb4q8\" (UniqueName: \"kubernetes.io/projected/0d2afc94-4048-4a25-b388-db8926951a27-kube-api-access-zb4q8\") pod \"cluster-proxy-proxy-agent-768868554d-w882x\" (UID: \"0d2afc94-4048-4a25-b388-db8926951a27\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.849438 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.849396 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw2qp\" (UniqueName: \"kubernetes.io/projected/f6995542-c356-4167-b9ab-0cf2c75d1c74-kube-api-access-zw2qp\") pod \"klusterlet-addon-workmgr-8d6d5b458-96z2c\" (UID: \"f6995542-c356-4167-b9ab-0cf2c75d1c74\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.922264 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.922178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:09.926750 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.926718 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:09.936686 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.936662 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" Jul 06 22:25:09.941047 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrgx\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-kube-api-access-kdrgx\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941161 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941065 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-bound-sa-token\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941161 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-registry-tls\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941161 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f330c5c0-7c8e-47b4-8924-7414a02f502e-image-registry-private-configuration\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f330c5c0-7c8e-47b4-8924-7414a02f502e-installation-pull-secrets\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941191 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f330c5c0-7c8e-47b4-8924-7414a02f502e-trusted-ca\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f330c5c0-7c8e-47b4-8924-7414a02f502e-ca-trust-extracted\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.941323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.941253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f330c5c0-7c8e-47b4-8924-7414a02f502e-registry-certificates\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.943240 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.942077 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f330c5c0-7c8e-47b4-8924-7414a02f502e-registry-certificates\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.943240 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.942379 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f330c5c0-7c8e-47b4-8924-7414a02f502e-ca-trust-extracted\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.943240 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.943049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f330c5c0-7c8e-47b4-8924-7414a02f502e-trusted-ca\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.944913 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.944891 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-registry-tls\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.945053 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.944996 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f330c5c0-7c8e-47b4-8924-7414a02f502e-installation-pull-secrets\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.946574 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.946537 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f330c5c0-7c8e-47b4-8924-7414a02f502e-image-registry-private-configuration\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.949668 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.949647 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrgx\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-kube-api-access-kdrgx\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:09.950197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:09.950179 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f330c5c0-7c8e-47b4-8924-7414a02f502e-bound-sa-token\") pod \"image-registry-7774bcc-qfphc\" (UID: \"f330c5c0-7c8e-47b4-8924-7414a02f502e\") " pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:10.022608 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.022534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:10.053325 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.053291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:10.077299 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.077232 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c"] Jul 06 22:25:10.081184 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:10.081144 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6995542_c356_4167_b9ab_0cf2c75d1c74.slice/crio-80ab17271e597944b9c6683a4e8b4e3f6ccba7888d72282d06e7a64da2737d04 WatchSource:0}: Error finding container 80ab17271e597944b9c6683a4e8b4e3f6ccba7888d72282d06e7a64da2737d04: Status 404 returned error can't find the container with id 80ab17271e597944b9c6683a4e8b4e3f6ccba7888d72282d06e7a64da2737d04 Jul 06 22:25:10.106001 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.105920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x"] Jul 06 22:25:10.108519 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.108493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:10.110413 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:10.110379 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d2afc94_4048_4a25_b388_db8926951a27.slice/crio-78b7a098c6e701e15429c657d947a3273116c2a814e77096f1c3d313e831b87b WatchSource:0}: Error finding container 78b7a098c6e701e15429c657d947a3273116c2a814e77096f1c3d313e831b87b: Status 404 returned error can't find the container with id 78b7a098c6e701e15429c657d947a3273116c2a814e77096f1c3d313e831b87b Jul 06 22:25:10.211230 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.211192 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7774bcc-qfphc"] Jul 06 22:25:10.214747 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:10.214704 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf330c5c0_7c8e_47b4_8924_7414a02f502e.slice/crio-413ec482ca6ee2a966170fff8201282ee81813bf35b64b02f4d71d0df8b38038 WatchSource:0}: Error finding container 413ec482ca6ee2a966170fff8201282ee81813bf35b64b02f4d71d0df8b38038: Status 404 returned error can't find the container with id 413ec482ca6ee2a966170fff8201282ee81813bf35b64b02f4d71d0df8b38038 Jul 06 22:25:10.324975 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.324936 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" event={"ID":"0d2afc94-4048-4a25-b388-db8926951a27","Type":"ContainerStarted","Data":"78b7a098c6e701e15429c657d947a3273116c2a814e77096f1c3d313e831b87b"} Jul 06 22:25:10.327243 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.327210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhpss" event={"ID":"8382f3d5-bbd9-4ce9-9b35-81a3debfa308","Type":"ContainerStarted","Data":"9a15d9122dbcc44bc8b7f188568c054bd977bdace97a0d794ddb7adf19ab0c9e"} Jul 06 22:25:10.329067 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.329041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774bcc-qfphc" event={"ID":"f330c5c0-7c8e-47b4-8924-7414a02f502e","Type":"ContainerStarted","Data":"23221955cd6d65fcef4d5c15f8f4dbe1a14a167ec398ab73dd6b372d64faaa87"} Jul 06 22:25:10.329189 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.329085 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7774bcc-qfphc" event={"ID":"f330c5c0-7c8e-47b4-8924-7414a02f502e","Type":"ContainerStarted","Data":"413ec482ca6ee2a966170fff8201282ee81813bf35b64b02f4d71d0df8b38038"} Jul 06 22:25:10.329189 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.329171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:10.330262 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.330242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" event={"ID":"f6995542-c356-4167-b9ab-0cf2c75d1c74","Type":"ContainerStarted","Data":"80ab17271e597944b9c6683a4e8b4e3f6ccba7888d72282d06e7a64da2737d04"} Jul 06 22:25:10.349462 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:10.349380 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7774bcc-qfphc" podStartSLOduration=1.349358397 podStartE2EDuration="1.349358397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:10.348675975 +0000 UTC m=+72.085077963" watchObservedRunningTime="2026-07-06 22:25:10.349358397 +0000 UTC m=+72.085760384" Jul 06 22:25:11.340629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:11.340474 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ftjg" event={"ID":"1689f60b-7849-452d-ba1a-27a8057595a9","Type":"ContainerStarted","Data":"d61c4178a734ebe13ac9a31b81dc1d5a0cedba0b994c501c5969b23799e0247f"} Jul 06 22:25:11.340629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:11.340528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ftjg" event={"ID":"1689f60b-7849-452d-ba1a-27a8057595a9","Type":"ContainerStarted","Data":"ea3429d09af00ab4086f0dcfd28c4ed53ad6a612e18ce1bc775240239bb8e6b8"} Jul 06 22:25:11.340629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:11.340576 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6ftjg" Jul 06 22:25:11.358922 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:11.357801 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6ftjg" podStartSLOduration=36.819610223 podStartE2EDuration="38.357779702s" totalImagesPullingTime="1.538169479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:24:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:11.356941786 +0000 UTC m=+73.093343775" watchObservedRunningTime="2026-07-06 22:25:11.357779702 +0000 UTC m=+73.094181691" Jul 06 22:25:15.247419 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:15.247370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:15.496165 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:15.496131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:15.567211 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:15.567121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:16.563765 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:16.563742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:16.662647 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:16.662594 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:16.732894 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:16.732861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:16.815406 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:16.815367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:17.095622 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.095589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:17.365002 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.364901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" event={"ID":"f6995542-c356-4167-b9ab-0cf2c75d1c74","Type":"ContainerStarted","Data":"ffeee87e1dce57eb7615b3e24f3d3d2301f1a447cd5c3cf4a4c3029791b1192f"} Jul 06 22:25:17.365002 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.364968 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:17.367007 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.366983 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" Jul 06 22:25:17.367324 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.367298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" event={"ID":"0d2afc94-4048-4a25-b388-db8926951a27","Type":"ContainerStarted","Data":"a6db2d9e23a17f6565e73d263e8b855d8a294acede0cb96deff9375851a18b02"} Jul 06 22:25:17.367457 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.367338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" event={"ID":"0d2afc94-4048-4a25-b388-db8926951a27","Type":"ContainerStarted","Data":"17d2f568c5afe67b979acc2299d79a3e382238b187284f82869359855d137999"} Jul 06 22:25:17.367457 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.367356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" event={"ID":"0d2afc94-4048-4a25-b388-db8926951a27","Type":"ContainerStarted","Data":"c5cc3546408af0214bfc3d625c25f296f0a59e8a4db8823a5a5a9c697bc8cfd0"} Jul 06 22:25:17.369327 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.369307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhpss" event={"ID":"8382f3d5-bbd9-4ce9-9b35-81a3debfa308","Type":"ContainerStarted","Data":"d52f79344f8774c4061bce9fd629504db3b3e61f714aaf264e1b20378ac8679c"} Jul 06 22:25:17.380766 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.380727 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8d6d5b458-96z2c" podStartSLOduration=1.9730751720000002 podStartE2EDuration="8.380714586s" totalImagesPullingTime="6.407639414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:17.379763145 +0000 UTC m=+79.116165144" watchObservedRunningTime="2026-07-06 22:25:17.380714586 +0000 UTC m=+79.117116572" Jul 06 22:25:17.396577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.396535 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bhpss" podStartSLOduration=17.180702314 podStartE2EDuration="30.396521773s" totalImagesPullingTime="13.215819459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 22:24:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:17.395214265 +0000 UTC m=+79.131616253" watchObservedRunningTime="2026-07-06 22:25:17.396521773 +0000 UTC m=+79.132923759" Jul 06 22:25:17.419279 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:17.416690 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768868554d-w882x" podStartSLOduration=2.038872476 podStartE2EDuration="8.416671837s" totalImagesPullingTime="6.377799361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:17.414568565 +0000 UTC m=+79.150970552" watchObservedRunningTime="2026-07-06 22:25:17.416671837 +0000 UTC m=+79.153073825" Jul 06 22:25:19.629878 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:19.629852 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:25:20.090294 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:20.090263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:20.191660 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:20.191631 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:20.707716 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:20.707675 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:20.772321 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:20.772289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:21.347060 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:21.347032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6ftjg" Jul 06 22:25:22.655062 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:22.655027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:22.712866 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:22.712840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:23.036925 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:23.036849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:23.137869 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:23.137836 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:24.245568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:24.245534 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:24.346065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:24.346033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:24.494288 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:24.494259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:24.599637 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:24.599607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:26.766622 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:26.766592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:26.846714 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:26.846680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:26.934672 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:26.934625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:28.297525 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:28.297493 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:28.398919 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:28.398891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:28.636119 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:28.636082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:29.134407 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:29.134374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:29.234672 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:29.234637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:29.695169 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:29.695116 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:29.795243 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:29.795211 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:29.913341 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:29.913311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.014245 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.014138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.121277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.121244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.228376 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.228325 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wt4nd"] Jul 06 22:25:30.229412 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.229382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.233167 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.233150 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.236135 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.236115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 22:25:30.237975 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.237961 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 22:25:30.238511 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.238491 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-svfcn\"" Jul 06 22:25:30.238511 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.238505 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 22:25:30.238648 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.238494 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 22:25:30.239129 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.239111 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 22:25:30.239216 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.239130 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 22:25:30.380929 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.380896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.403025 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.402995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-textfile\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcsd\" (UniqueName: \"kubernetes.io/projected/fbe9c034-1f93-4690-9202-4e72ff368100-kube-api-access-rdcsd\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403147 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-sys\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403313 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403174 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbe9c034-1f93-4690-9202-4e72ff368100-metrics-client-ca\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403313 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403209 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-wtmp\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403313 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-tls\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.403403 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.403320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-root\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.461325 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.461275 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.504514 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-textfile\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcsd\" (UniqueName: \"kubernetes.io/projected/fbe9c034-1f93-4690-9202-4e72ff368100-kube-api-access-rdcsd\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-sys\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbe9c034-1f93-4690-9202-4e72ff368100-metrics-client-ca\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-wtmp\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504824 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-sys\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-tls\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.504887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-root\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.505073 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504960 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-root\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.505073 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.504960 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-wtmp\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.505243 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.505221 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-accelerators-collector-config\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.505315 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.505251 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-textfile\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.505315 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.505294 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fbe9c034-1f93-4690-9202-4e72ff368100-metrics-client-ca\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.506979 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.506945 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.507092 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.507063 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fbe9c034-1f93-4690-9202-4e72ff368100-node-exporter-tls\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.515039 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.515016 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcsd\" (UniqueName: \"kubernetes.io/projected/fbe9c034-1f93-4690-9202-4e72ff368100-kube-api-access-rdcsd\") pod \"node-exporter-wt4nd\" (UID: \"fbe9c034-1f93-4690-9202-4e72ff368100\") " pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.544725 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.544693 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wt4nd" Jul 06 22:25:30.559148 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:25:30.559080 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe9c034_1f93_4690_9202_4e72ff368100.slice/crio-77da79f83b66508c8205ca294a7efb958f64488151eb85a8d50a84442a0be833 WatchSource:0}: Error finding container 77da79f83b66508c8205ca294a7efb958f64488151eb85a8d50a84442a0be833: Status 404 returned error can't find the container with id 77da79f83b66508c8205ca294a7efb958f64488151eb85a8d50a84442a0be833 Jul 06 22:25:30.563120 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.563092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.661612 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.661531 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:30.732842 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:30.732807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:31.085399 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.085366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:31.172944 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.172911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:31.346396 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.346325 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7774bcc-qfphc" Jul 06 22:25:31.418957 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.418912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt4nd" event={"ID":"fbe9c034-1f93-4690-9202-4e72ff368100","Type":"ContainerStarted","Data":"77da79f83b66508c8205ca294a7efb958f64488151eb85a8d50a84442a0be833"} Jul 06 22:25:31.551254 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.551229 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:31.655690 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.655659 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:31.852044 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:31.851963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:32.103642 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:32.103549 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:32.205215 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:32.205145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:32.423663 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:32.423580 2564 generic.go:356] "Generic (PLEG): container finished" podID="fbe9c034-1f93-4690-9202-4e72ff368100" containerID="04b014b294de3506952ae4a2d48d5a3ea7a1d98fd64ea8f21c0598e12a5bc574" exitCode=0 Jul 06 22:25:32.423663 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:32.423622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt4nd" event={"ID":"fbe9c034-1f93-4690-9202-4e72ff368100","Type":"ContainerDied","Data":"04b014b294de3506952ae4a2d48d5a3ea7a1d98fd64ea8f21c0598e12a5bc574"} Jul 06 22:25:32.817542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:32.817471 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:32.917620 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:32.917589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.002886 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.002843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.112692 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.112660 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.218473 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.218437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.311543 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.311510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.429751 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.429675 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt4nd" event={"ID":"fbe9c034-1f93-4690-9202-4e72ff368100","Type":"ContainerStarted","Data":"9d98b47d485128d52aac30c2dd9a3ba07d7f8210706680bba001dab057008f7c"} Jul 06 22:25:33.429751 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.429724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wt4nd" event={"ID":"fbe9c034-1f93-4690-9202-4e72ff368100","Type":"ContainerStarted","Data":"c69127b3ecaf857af232e01b9e1f610bf95886dab409caf0d2a51e2ed3944617"} Jul 06 22:25:33.449540 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.449500 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wt4nd" podStartSLOduration=2.402287457 podStartE2EDuration="3.449484698s" totalImagesPullingTime="982.543356ms" totalInitContainerRuntime="64.653885ms" isStatefulPod=true podCreationTimestamp="2026-07-06 22:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:25:33.448276079 +0000 UTC m=+95.184678066" watchObservedRunningTime="2026-07-06 22:25:33.449484698 +0000 UTC m=+95.185886685" Jul 06 22:25:33.578303 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.578265 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.680568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.680477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:33.806642 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:33.806606 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:34.281952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.281918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:34.351487 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.351452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:34.651037 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.650997 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" podUID="a00d43cd-f3ab-41f1-b464-641c2fc88979" containerName="registry" containerID="cri-o://24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8" gracePeriod=30 Jul 06 22:25:34.903755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.903692 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:25:34.944917 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.944885 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:34.946855 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.946835 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-bound-sa-token\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.946951 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.946869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-image-registry-private-configuration\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.946951 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.946899 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a00d43cd-f3ab-41f1-b464-641c2fc88979-ca-trust-extracted\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.946951 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.946931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.947093 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.946960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-installation-pull-secrets\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.947093 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.946992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-certificates\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.947937 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.947677 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:25:34.948132 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.948112 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-trusted-ca\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.948724 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.948707 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppk8b\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-kube-api-access-ppk8b\") pod \"a00d43cd-f3ab-41f1-b464-641c2fc88979\" (UID: \"a00d43cd-f3ab-41f1-b464-641c2fc88979\") " Jul 06 22:25:34.949070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.949050 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-certificates\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:34.950065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.948650 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:25:34.950163 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.950138 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:25:34.951754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.951714 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:25:34.951754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.951727 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:25:34.952080 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.952043 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:25:34.952163 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.952091 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-kube-api-access-ppk8b" (OuterVolumeSpecName: "kube-api-access-ppk8b") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "kube-api-access-ppk8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:25:34.958408 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:34.958383 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00d43cd-f3ab-41f1-b464-641c2fc88979-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a00d43cd-f3ab-41f1-b464-641c2fc88979" (UID: "a00d43cd-f3ab-41f1-b464-641c2fc88979"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:25:35.033989 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.033956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:35.051181 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051156 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a00d43cd-f3ab-41f1-b464-641c2fc88979-trusted-ca\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.051181 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051180 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppk8b\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-kube-api-access-ppk8b\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.051323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051191 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-bound-sa-token\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.051323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051201 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-image-registry-private-configuration\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.051323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051210 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a00d43cd-f3ab-41f1-b464-641c2fc88979-ca-trust-extracted\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.051323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051218 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a00d43cd-f3ab-41f1-b464-641c2fc88979-registry-tls\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.051323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.051226 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a00d43cd-f3ab-41f1-b464-641c2fc88979-installation-pull-secrets\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:25:35.248452 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.248301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:35.438580 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.438532 2564 generic.go:356] "Generic (PLEG): container finished" podID="a00d43cd-f3ab-41f1-b464-641c2fc88979" containerID="24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8" exitCode=0 Jul 06 22:25:35.438580 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.438569 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" event={"ID":"a00d43cd-f3ab-41f1-b464-641c2fc88979","Type":"ContainerDied","Data":"24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8"} Jul 06 22:25:35.439103 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.438588 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" Jul 06 22:25:35.439103 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.438623 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66599dd4f7-vqp4q" event={"ID":"a00d43cd-f3ab-41f1-b464-641c2fc88979","Type":"ContainerDied","Data":"8f89661b064dd4f4bd1cf20fff03bcf42712226fc6c2004a23cfca57944804e0"} Jul 06 22:25:35.439103 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.438646 2564 scope.go:117] "RemoveContainer" containerID="24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8" Jul 06 22:25:35.447670 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.447653 2564 scope.go:117] "RemoveContainer" containerID="24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8" Jul 06 22:25:35.447915 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:25:35.447894 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8\": container with ID starting with 24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8 not found: ID does not exist" containerID="24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8" Jul 06 22:25:35.447982 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.447922 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8"} err="failed to get container status \"24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8\": rpc error: code = NotFound desc = could not find container \"24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8\": container with ID starting with 24cc4edf939cab28b26b14e8db2033474924f02c6b2752196881ed7e685e99e8 not found: ID does not exist" Jul 06 22:25:35.462354 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.462331 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66599dd4f7-vqp4q"] Jul 06 22:25:35.469163 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.469144 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66599dd4f7-vqp4q"] Jul 06 22:25:35.826292 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.826258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:35.927331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:35.927297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:36.030984 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:36.030950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:36.301377 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:36.301345 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-j25q7" Jul 06 22:25:36.452531 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:36.452502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:36.809704 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:36.809674 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a00d43cd-f3ab-41f1-b464-641c2fc88979" path="/var/lib/kubelet/pods/a00d43cd-f3ab-41f1-b464-641c2fc88979/volumes" Jul 06 22:25:43.769553 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:43.769517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:44.326556 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:44.326519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:44.427583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:44.427553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:44.778890 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:44.778843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:44.929031 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:44.928996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:45.112767 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:45.112734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:46.349273 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:46.349235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:46.448779 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:46.448750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:47.520244 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:47.520204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:47.620459 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:47.620409 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:49.182474 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:49.182419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:49.282942 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:49.282893 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:54.994641 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:54.994592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:55.095571 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:55.095536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:25:55.196811 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:25:55.196775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:02.542184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:02.542147 2564 generic.go:356] "Generic (PLEG): container finished" podID="1f4aca1d-2fa0-475b-bac8-b5903a9e68b1" containerID="1d4a05076cca7fe10a4303db8e36078c68116d42a8aa1de32f60c7e93a87df5f" exitCode=0 Jul 06 22:26:02.542599 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:02.542217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" event={"ID":"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1","Type":"ContainerDied","Data":"1d4a05076cca7fe10a4303db8e36078c68116d42a8aa1de32f60c7e93a87df5f"} Jul 06 22:26:02.542599 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:02.542558 2564 scope.go:117] "RemoveContainer" containerID="1d4a05076cca7fe10a4303db8e36078c68116d42a8aa1de32f60c7e93a87df5f" Jul 06 22:26:03.547304 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:03.547270 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-k67cf" event={"ID":"1f4aca1d-2fa0-475b-bac8-b5903a9e68b1","Type":"ContainerStarted","Data":"bac8bb63c953bf95a4148696a9905ce2e48996f56a6676d8bb622b2df47de391"} Jul 06 22:26:04.195690 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:04.195656 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:04.291660 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:04.291616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:04.392048 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:04.392008 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:19.182322 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:19.182272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:19.253478 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:19.253420 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:26.675160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:26.675105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:46.255696 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:46.255650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:46.309259 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:46.309230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:51.232451 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:51.232366 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:51.361409 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:51.361374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:51.462187 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:51.462160 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:51.969718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:51.969687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:52.070331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:52.070306 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:58.186283 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:58.186234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:26:58.241550 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:26:58.241519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:02.319921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:02.319864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:02.420881 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:02.420845 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:02.525961 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:02.525931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:04.185574 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:04.185541 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:04.285357 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:04.285327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:18.918517 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:18.918486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:31.279185 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:31.279151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:32.620141 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:32.620104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:44.538831 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:44.538774 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:27:44.636709 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:27:44.636677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:07.654606 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:07.654570 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:10.958695 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:10.958655 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:12.369310 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:12.369272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:14.562977 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:14.562944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:14.618867 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:14.618833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:15.614032 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:15.613993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:15.724031 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:15.723998 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:16.555871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:16.555838 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:16.640408 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:16.640376 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:18.776756 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:18.776722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:18.969600 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:18.969565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:21.765373 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:21.765287 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:21.821894 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:21.821863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:26.999547 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:26.999511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:27.785502 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:27.785466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:27.883329 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:27.883302 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:34.221730 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:34.221695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:34.322465 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:34.322415 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:35.938851 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:35.938812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:39.648024 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:39.647988 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:39.749671 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:39.749640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:46.682029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:46.681979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:48.988665 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:48.988629 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:49.086185 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:49.086149 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:49.636182 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:49.636124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:49.875023 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:49.874992 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:49.983409 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:49.983300 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:50.026989 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.026954 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b"] Jul 06 22:28:50.027508 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.027347 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a00d43cd-f3ab-41f1-b464-641c2fc88979" containerName="registry" Jul 06 22:28:50.027508 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.027366 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a00d43cd-f3ab-41f1-b464-641c2fc88979" containerName="registry" Jul 06 22:28:50.027508 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.027450 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a00d43cd-f3ab-41f1-b464-641c2fc88979" containerName="registry" Jul 06 22:28:50.030439 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.030406 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.034206 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.034184 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 22:28:50.034339 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.034255 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-n44fs\"" Jul 06 22:28:50.034395 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.034342 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 22:28:50.034823 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.034805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 22:28:50.037467 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.037446 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 22:28:50.049307 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.049283 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b"] Jul 06 22:28:50.127385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.127355 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbd4a953-28be-49c0-a00f-9508dc0b595e-tmp\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.127385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.127386 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbd4a953-28be-49c0-a00f-9508dc0b595e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.127602 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.127410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbd4a953-28be-49c0-a00f-9508dc0b595e-manifests\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.127602 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.127463 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkhq9\" (UniqueName: \"kubernetes.io/projected/cbd4a953-28be-49c0-a00f-9508dc0b595e-kube-api-access-zkhq9\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.127602 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.127525 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbd4a953-28be-49c0-a00f-9508dc0b595e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.164504 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.164473 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:50.228774 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.228749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbd4a953-28be-49c0-a00f-9508dc0b595e-tmp\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.228899 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.228775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbd4a953-28be-49c0-a00f-9508dc0b595e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.228899 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.228795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbd4a953-28be-49c0-a00f-9508dc0b595e-manifests\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.228899 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.228883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkhq9\" (UniqueName: \"kubernetes.io/projected/cbd4a953-28be-49c0-a00f-9508dc0b595e-kube-api-access-zkhq9\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.229036 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.228922 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbd4a953-28be-49c0-a00f-9508dc0b595e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.229605 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.229576 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cbd4a953-28be-49c0-a00f-9508dc0b595e-manifests\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.232376 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.232269 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cbd4a953-28be-49c0-a00f-9508dc0b595e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.232608 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.232583 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cbd4a953-28be-49c0-a00f-9508dc0b595e-tmp\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.235814 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.235758 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cbd4a953-28be-49c0-a00f-9508dc0b595e-webhook-cert\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.244950 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.244923 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkhq9\" (UniqueName: \"kubernetes.io/projected/cbd4a953-28be-49c0-a00f-9508dc0b595e-kube-api-access-zkhq9\") pod \"opendatahub-operator-controller-manager-84b8787b6c-kc67b\" (UID: \"cbd4a953-28be-49c0-a00f-9508dc0b595e\") " pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.277893 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.277862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:50.341084 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.341064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:50.414441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.414397 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:50.497959 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.497889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b"] Jul 06 22:28:50.501853 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:28:50.501830 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd4a953_28be_49c0_a00f_9508dc0b595e.slice/crio-70315e716271b061efb1a244e12ed718d29746cfdada1e7cd9c29bdc8fa88219 WatchSource:0}: Error finding container 70315e716271b061efb1a244e12ed718d29746cfdada1e7cd9c29bdc8fa88219: Status 404 returned error can't find the container with id 70315e716271b061efb1a244e12ed718d29746cfdada1e7cd9c29bdc8fa88219 Jul 06 22:28:50.541153 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.541126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:50.627968 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.627942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:50.729590 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:50.729545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:51.145445 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:51.145388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" event={"ID":"cbd4a953-28be-49c0-a00f-9508dc0b595e","Type":"ContainerStarted","Data":"70315e716271b061efb1a244e12ed718d29746cfdada1e7cd9c29bdc8fa88219"} Jul 06 22:28:54.160042 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:54.160008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" event={"ID":"cbd4a953-28be-49c0-a00f-9508dc0b595e","Type":"ContainerStarted","Data":"64840d1fca14f9ca254cd5a612e8a53d7a0c5bb027b83243de3eb5a5265542e9"} Jul 06 22:28:55.164882 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:55.164847 2564 generic.go:356] "Generic (PLEG): container finished" podID="cbd4a953-28be-49c0-a00f-9508dc0b595e" containerID="64840d1fca14f9ca254cd5a612e8a53d7a0c5bb027b83243de3eb5a5265542e9" exitCode=0 Jul 06 22:28:55.165362 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:55.164890 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" event={"ID":"cbd4a953-28be-49c0-a00f-9508dc0b595e","Type":"ContainerDied","Data":"64840d1fca14f9ca254cd5a612e8a53d7a0c5bb027b83243de3eb5a5265542e9"} Jul 06 22:28:56.170667 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:56.170627 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" event={"ID":"cbd4a953-28be-49c0-a00f-9508dc0b595e","Type":"ContainerStarted","Data":"c2775d4dd7643dc161c47356035250fd9ae0234d61fe66e3b101281a41008260"} Jul 06 22:28:56.171049 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:56.170720 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:28:56.192930 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:56.192872 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" podStartSLOduration=2.0806265919999998 podStartE2EDuration="6.192854667s" totalImagesPullingTime="3.076590781s" totalInitContainerRuntime="1.035637294s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:28:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:28:56.190717131 +0000 UTC m=+297.927119118" watchObservedRunningTime="2026-07-06 22:28:56.192854667 +0000 UTC m=+297.929256654" Jul 06 22:28:58.151338 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:58.151298 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:58.337204 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:58.337165 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:28:58.649958 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:58.649929 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:28:58.656297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:58.656273 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 22:28:58.667795 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:28:58.667770 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:29:01.279900 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:01.279862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:01.378844 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:01.378806 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:07.177083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:07.177055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-84b8787b6c-kc67b" Jul 06 22:29:09.333553 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:09.333519 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:09.506852 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:09.506817 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.125414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.125372 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm"] Jul 06 22:29:10.131919 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.131894 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.134539 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.134511 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 22:29:10.134539 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.134530 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 22:29:10.134732 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.134585 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-tg4wb\"" Jul 06 22:29:10.135542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.135522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.151316 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.151294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm"] Jul 06 22:29:10.200759 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.200726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3157ed2-0dff-4199-8764-478bc8e591db-tmp\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.200894 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.200795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3157ed2-0dff-4199-8764-478bc8e591db-tls-certs\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.200894 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.200830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w5v\" (UniqueName: \"kubernetes.io/projected/a3157ed2-0dff-4199-8764-478bc8e591db-kube-api-access-r6w5v\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.301549 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.301511 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w5v\" (UniqueName: \"kubernetes.io/projected/a3157ed2-0dff-4199-8764-478bc8e591db-kube-api-access-r6w5v\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.301793 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.301577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3157ed2-0dff-4199-8764-478bc8e591db-tmp\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.301793 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.301622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3157ed2-0dff-4199-8764-478bc8e591db-tls-certs\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.303804 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.303774 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3157ed2-0dff-4199-8764-478bc8e591db-tmp\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.304158 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.304129 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3157ed2-0dff-4199-8764-478bc8e591db-tls-certs\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.334321 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.334289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w5v\" (UniqueName: \"kubernetes.io/projected/a3157ed2-0dff-4199-8764-478bc8e591db-kube-api-access-r6w5v\") pod \"kube-auth-proxy-7c6b7b4fbb-6t9nm\" (UID: \"a3157ed2-0dff-4199-8764-478bc8e591db\") " pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.356110 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.356079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.416567 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.416486 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2dkjj"] Jul 06 22:29:10.418855 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.418828 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:10.422137 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.422111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 22:29:10.422269 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.422119 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2nvcv\"" Jul 06 22:29:10.439099 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.439071 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2dkjj"] Jul 06 22:29:10.441454 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.441417 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" Jul 06 22:29:10.482397 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.482359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.503657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.503542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7m8d\" (UniqueName: \"kubernetes.io/projected/5b717620-cdd9-4e9d-bc89-7280bf8108fe-kube-api-access-c7m8d\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:10.503657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.503628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b717620-cdd9-4e9d-bc89-7280bf8108fe-cert\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:10.556233 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.556179 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.589955 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.589934 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm"] Jul 06 22:29:10.593036 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:29:10.593012 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3157ed2_0dff_4199_8764_478bc8e591db.slice/crio-9e8568907b82381b676c477ec029b2af2a25188850016b5a3527967c3767971f WatchSource:0}: Error finding container 9e8568907b82381b676c477ec029b2af2a25188850016b5a3527967c3767971f: Status 404 returned error can't find the container with id 9e8568907b82381b676c477ec029b2af2a25188850016b5a3527967c3767971f Jul 06 22:29:10.594897 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.594880 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:29:10.604662 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.604640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b717620-cdd9-4e9d-bc89-7280bf8108fe-cert\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:10.604730 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.604692 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7m8d\" (UniqueName: \"kubernetes.io/projected/5b717620-cdd9-4e9d-bc89-7280bf8108fe-kube-api-access-c7m8d\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:10.604830 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:29:10.604811 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 22:29:10.604895 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:29:10.604885 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b717620-cdd9-4e9d-bc89-7280bf8108fe-cert podName:5b717620-cdd9-4e9d-bc89-7280bf8108fe nodeName:}" failed. No retries permitted until 2026-07-06 22:29:11.10486406 +0000 UTC m=+312.841266029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5b717620-cdd9-4e9d-bc89-7280bf8108fe-cert") pod "odh-model-controller-7998c67dc4-2dkjj" (UID: "5b717620-cdd9-4e9d-bc89-7280bf8108fe") : secret "odh-model-controller-webhook-cert" not found Jul 06 22:29:10.617703 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.617681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7m8d\" (UniqueName: \"kubernetes.io/projected/5b717620-cdd9-4e9d-bc89-7280bf8108fe-kube-api-access-c7m8d\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:10.648132 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.648103 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:10.727486 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:10.727390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:11.108606 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.108568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b717620-cdd9-4e9d-bc89-7280bf8108fe-cert\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:11.110983 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.110963 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b717620-cdd9-4e9d-bc89-7280bf8108fe-cert\") pod \"odh-model-controller-7998c67dc4-2dkjj\" (UID: \"5b717620-cdd9-4e9d-bc89-7280bf8108fe\") " pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:11.235655 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.235614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" event={"ID":"a3157ed2-0dff-4199-8764-478bc8e591db","Type":"ContainerStarted","Data":"9e8568907b82381b676c477ec029b2af2a25188850016b5a3527967c3767971f"} Jul 06 22:29:11.330110 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.330068 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:11.506354 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.506313 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6c44h"] Jul 06 22:29:11.509256 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.509226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.518644 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.518618 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 22:29:11.518644 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.518635 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nn6j7\"" Jul 06 22:29:11.531871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.531718 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6c44h"] Jul 06 22:29:11.558306 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.558272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:11.612779 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.612735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b9b2b7-b744-4ed2-a6ca-cd4976cec84c-tls-certs\") pod \"model-serving-api-54d89bd5fc-6c44h\" (UID: \"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.612944 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.612791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn9p2\" (UniqueName: \"kubernetes.io/projected/17b9b2b7-b744-4ed2-a6ca-cd4976cec84c-kube-api-access-bn9p2\") pod \"model-serving-api-54d89bd5fc-6c44h\" (UID: \"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.688364 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.688286 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-2dkjj"] Jul 06 22:29:11.692232 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:29:11.692201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b717620_cdd9_4e9d_bc89_7280bf8108fe.slice/crio-8587dbeedb1b4ee9443e02103c24da54d823bb5c6d66455e44d8a29dd3f31d81 WatchSource:0}: Error finding container 8587dbeedb1b4ee9443e02103c24da54d823bb5c6d66455e44d8a29dd3f31d81: Status 404 returned error can't find the container with id 8587dbeedb1b4ee9443e02103c24da54d823bb5c6d66455e44d8a29dd3f31d81 Jul 06 22:29:11.692651 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.692569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:11.713583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.713552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b9b2b7-b744-4ed2-a6ca-cd4976cec84c-tls-certs\") pod \"model-serving-api-54d89bd5fc-6c44h\" (UID: \"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.713746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.713602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn9p2\" (UniqueName: \"kubernetes.io/projected/17b9b2b7-b744-4ed2-a6ca-cd4976cec84c-kube-api-access-bn9p2\") pod \"model-serving-api-54d89bd5fc-6c44h\" (UID: \"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.716268 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.716238 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17b9b2b7-b744-4ed2-a6ca-cd4976cec84c-tls-certs\") pod \"model-serving-api-54d89bd5fc-6c44h\" (UID: \"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.725741 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.725717 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn9p2\" (UniqueName: \"kubernetes.io/projected/17b9b2b7-b744-4ed2-a6ca-cd4976cec84c-kube-api-access-bn9p2\") pod \"model-serving-api-54d89bd5fc-6c44h\" (UID: \"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.823609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.823569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:11.886459 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.884529 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:11.886459 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.886322 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-278f7"] Jul 06 22:29:11.891857 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.891829 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:11.894496 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.894472 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-db6bf\"" Jul 06 22:29:11.894629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.894552 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 22:29:11.894629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.894570 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 22:29:11.894828 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.894810 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 22:29:11.894966 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.894940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 22:29:11.898063 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.898036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-278f7"] Jul 06 22:29:11.963471 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.963359 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:11.976255 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:11.976232 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6c44h"] Jul 06 22:29:11.978675 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:29:11.978650 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b9b2b7_b744_4ed2_a6ca_cd4976cec84c.slice/crio-89f4f1da4053c7868e5122b41efac2ced9112949090c885544efdac722a31067 WatchSource:0}: Error finding container 89f4f1da4053c7868e5122b41efac2ced9112949090c885544efdac722a31067: Status 404 returned error can't find the container with id 89f4f1da4053c7868e5122b41efac2ced9112949090c885544efdac722a31067 Jul 06 22:29:12.016668 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bb2q\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-kube-api-access-4bb2q\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.016804 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-cacerts\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.016804 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016744 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.016804 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016763 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/792f40ca-011a-4c34-bf8b-830f0b5f230c-local-certs\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.016908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.016908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-token\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.016908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.016896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.117728 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.117691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-cacerts\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.117931 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.117743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.117931 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.117770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/792f40ca-011a-4c34-bf8b-830f0b5f230c-local-certs\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.117931 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.117809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.118072 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.117933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-token\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.118072 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.117984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.118072 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.118020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bb2q\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-kube-api-access-4bb2q\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.118531 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.118476 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.120445 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.120400 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/792f40ca-011a-4c34-bf8b-830f0b5f230c-local-certs\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.120697 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.120667 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.120811 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.120710 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-cacerts\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.120970 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.120950 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.126014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.125990 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-token\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.126527 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.126510 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bb2q\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-kube-api-access-4bb2q\") pod \"istiod-openshift-gateway-7574459656-278f7\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.206411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.206372 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:12.242222 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.242179 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" event={"ID":"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c","Type":"ContainerStarted","Data":"89f4f1da4053c7868e5122b41efac2ced9112949090c885544efdac722a31067"} Jul 06 22:29:12.243496 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.243465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" event={"ID":"5b717620-cdd9-4e9d-bc89-7280bf8108fe","Type":"ContainerStarted","Data":"8587dbeedb1b4ee9443e02103c24da54d823bb5c6d66455e44d8a29dd3f31d81"} Jul 06 22:29:12.375982 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:12.375903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-278f7"] Jul 06 22:29:12.380904 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:29:12.380872 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792f40ca_011a_4c34_bf8b_830f0b5f230c.slice/crio-ab7d93a881132291c55a51ec804a3cf26e30ec0c883160e7d8443d2869a6a86c WatchSource:0}: Error finding container ab7d93a881132291c55a51ec804a3cf26e30ec0c883160e7d8443d2869a6a86c: Status 404 returned error can't find the container with id ab7d93a881132291c55a51ec804a3cf26e30ec0c883160e7d8443d2869a6a86c Jul 06 22:29:13.161479 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:13.161441 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:13.251646 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:13.251568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" event={"ID":"792f40ca-011a-4c34-bf8b-830f0b5f230c","Type":"ContainerStarted","Data":"ab7d93a881132291c55a51ec804a3cf26e30ec0c883160e7d8443d2869a6a86c"} Jul 06 22:29:16.441338 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:16.441297 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:29:16.441745 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:16.441385 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:29:17.274876 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.274841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" event={"ID":"a3157ed2-0dff-4199-8764-478bc8e591db","Type":"ContainerStarted","Data":"ba287ac1a811ac08b1303a1d7eecb19f408d596c84f276ac0f7820172b14511f"} Jul 06 22:29:17.276655 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.276622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" event={"ID":"17b9b2b7-b744-4ed2-a6ca-cd4976cec84c","Type":"ContainerStarted","Data":"8d8359ac3e5fa675eeec16c7a94d010f37876bc310c1105ff559a3e249a95ef7"} Jul 06 22:29:17.276810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.276721 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:17.278216 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.278189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" event={"ID":"5b717620-cdd9-4e9d-bc89-7280bf8108fe","Type":"ContainerStarted","Data":"57fecb8254de9527d2d16ec348bacaad2883b36947cea76553107c7f448f33c5"} Jul 06 22:29:17.278358 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.278258 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:17.279803 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.279782 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" event={"ID":"792f40ca-011a-4c34-bf8b-830f0b5f230c","Type":"ContainerStarted","Data":"3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f"} Jul 06 22:29:17.279998 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.279982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:17.281626 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.281599 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-278f7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 22:29:17.281707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.281650 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 22:29:17.322985 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.322929 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c6b7b4fbb-6t9nm" podStartSLOduration=2.417253455 podStartE2EDuration="7.322912084s" totalImagesPullingTime="4.905658629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:17.32115413 +0000 UTC m=+319.057556117" watchObservedRunningTime="2026-07-06 22:29:17.322912084 +0000 UTC m=+319.059314070" Jul 06 22:29:17.419312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.419242 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" podStartSLOduration=2.899115161 podStartE2EDuration="6.419220592s" totalImagesPullingTime="3.520105431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:17.363518426 +0000 UTC m=+319.099920412" watchObservedRunningTime="2026-07-06 22:29:17.419220592 +0000 UTC m=+319.155622581" Jul 06 22:29:17.420014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.419984 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" podStartSLOduration=2.7243147580000002 podStartE2EDuration="7.419974871s" totalImagesPullingTime="4.695660113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:17.418560817 +0000 UTC m=+319.154962804" watchObservedRunningTime="2026-07-06 22:29:17.419974871 +0000 UTC m=+319.156376859" Jul 06 22:29:17.458765 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.458699 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" podStartSLOduration=2.400905396 podStartE2EDuration="6.458684644s" totalImagesPullingTime="4.057779248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:17.458666536 +0000 UTC m=+319.195068524" watchObservedRunningTime="2026-07-06 22:29:17.458684644 +0000 UTC m=+319.195086633" Jul 06 22:29:17.612392 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.612344 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:17.812110 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:17.812075 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:18.284755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.284723 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-278f7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 22:29:18.284904 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.284801 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 22:29:18.525001 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.524962 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j"] Jul 06 22:29:18.528555 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.528532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.532085 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.532058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-b74q7\"" Jul 06 22:29:18.566555 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.566529 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j"] Jul 06 22:29:18.684059 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684059 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684074 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684098 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrjr\" (UniqueName: \"kubernetes.io/projected/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-kube-api-access-lqrjr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684212 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.684474 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.684287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.784949 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.784916 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.784949 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.784951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.784982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrjr\" (UniqueName: \"kubernetes.io/projected/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-kube-api-access-lqrjr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785223 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785742 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785708 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785864 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785772 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785864 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785813 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785864 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785817 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.785864 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.785859 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.787485 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.787467 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.787729 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.787711 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.793717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.793688 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.797017 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.796996 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrjr\" (UniqueName: \"kubernetes.io/projected/b830ff70-f9c1-45be-b0b8-b2fcc42bb772-kube-api-access-lqrjr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4vkz5j\" (UID: \"b830ff70-f9c1-45be-b0b8-b2fcc42bb772\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.839710 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.839344 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:18.938716 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:18.938676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:19.000734 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:19.000709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j"] Jul 06 22:29:19.003252 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:29:19.003227 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb830ff70_f9c1_45be_b0b8_b2fcc42bb772.slice/crio-d2d1f0a153a1a9053b61555a5a8af44abb2d9da20ff9f1b9ec27b6f7cda18fcb WatchSource:0}: Error finding container d2d1f0a153a1a9053b61555a5a8af44abb2d9da20ff9f1b9ec27b6f7cda18fcb: Status 404 returned error can't find the container with id d2d1f0a153a1a9053b61555a5a8af44abb2d9da20ff9f1b9ec27b6f7cda18fcb Jul 06 22:29:19.288738 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:19.288699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" event={"ID":"b830ff70-f9c1-45be-b0b8-b2fcc42bb772","Type":"ContainerStarted","Data":"d2d1f0a153a1a9053b61555a5a8af44abb2d9da20ff9f1b9ec27b6f7cda18fcb"} Jul 06 22:29:21.284953 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.284897 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:21.571742 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.571650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:21.649205 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.649170 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xthph"] Jul 06 22:29:21.652704 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.652683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.656224 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.656202 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qgj7p\"" Jul 06 22:29:21.657215 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.657191 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 22:29:21.670550 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.670523 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xthph"] Jul 06 22:29:21.709154 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.709119 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:21.808655 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.808621 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smsqb\" (UniqueName: \"kubernetes.io/projected/4e0a57f9-b003-45f9-86a9-b0752d1aede5-kube-api-access-smsqb\") pod \"kserve-controller-manager-d6f4b65b8-xthph\" (UID: \"4e0a57f9-b003-45f9-86a9-b0752d1aede5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.808844 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.808710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e0a57f9-b003-45f9-86a9-b0752d1aede5-cert\") pod \"kserve-controller-manager-d6f4b65b8-xthph\" (UID: \"4e0a57f9-b003-45f9-86a9-b0752d1aede5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.909604 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.909567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e0a57f9-b003-45f9-86a9-b0752d1aede5-cert\") pod \"kserve-controller-manager-d6f4b65b8-xthph\" (UID: \"4e0a57f9-b003-45f9-86a9-b0752d1aede5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.909789 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.909651 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smsqb\" (UniqueName: \"kubernetes.io/projected/4e0a57f9-b003-45f9-86a9-b0752d1aede5-kube-api-access-smsqb\") pod \"kserve-controller-manager-d6f4b65b8-xthph\" (UID: \"4e0a57f9-b003-45f9-86a9-b0752d1aede5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.912676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.912645 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e0a57f9-b003-45f9-86a9-b0752d1aede5-cert\") pod \"kserve-controller-manager-d6f4b65b8-xthph\" (UID: \"4e0a57f9-b003-45f9-86a9-b0752d1aede5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.921009 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.920977 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smsqb\" (UniqueName: \"kubernetes.io/projected/4e0a57f9-b003-45f9-86a9-b0752d1aede5-kube-api-access-smsqb\") pod \"kserve-controller-manager-d6f4b65b8-xthph\" (UID: \"4e0a57f9-b003-45f9-86a9-b0752d1aede5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.967817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.967778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:21.973721 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:21.973469 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:22.130354 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:22.130282 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:22.828076 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:22.828040 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:29:22.828359 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:22.828115 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:29:22.828359 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:22.828148 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:29:22.905942 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:22.905918 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xthph"] Jul 06 22:29:22.908494 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:29:22.908473 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e0a57f9_b003_45f9_86a9_b0752d1aede5.slice/crio-f448321e92ccbf78113af0af753862cdb70453559f89790cff2af3751ce57466 WatchSource:0}: Error finding container f448321e92ccbf78113af0af753862cdb70453559f89790cff2af3751ce57466: Status 404 returned error can't find the container with id f448321e92ccbf78113af0af753862cdb70453559f89790cff2af3751ce57466 Jul 06 22:29:22.977806 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:22.977748 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:23.108304 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.108268 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:23.237707 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.237674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:29:23.308152 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.308109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" event={"ID":"b830ff70-f9c1-45be-b0b8-b2fcc42bb772","Type":"ContainerStarted","Data":"5455dfba6094a6e1047bff651a46b1cdb862f2a9a0ad53db09ed895651fa1ca0"} Jul 06 22:29:23.309294 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.309269 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" event={"ID":"4e0a57f9-b003-45f9-86a9-b0752d1aede5","Type":"ContainerStarted","Data":"f448321e92ccbf78113af0af753862cdb70453559f89790cff2af3751ce57466"} Jul 06 22:29:23.357263 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.357206 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" podStartSLOduration=1.5347290139999998 podStartE2EDuration="5.357189226s" totalImagesPullingTime="3.822460212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:23.356171892 +0000 UTC m=+325.092573891" watchObservedRunningTime="2026-07-06 22:29:23.357189226 +0000 UTC m=+325.093591212" Jul 06 22:29:23.723897 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.723858 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-278f7"] Jul 06 22:29:23.724078 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.724044 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerName="discovery" containerID="cri-o://3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f" gracePeriod=30 Jul 06 22:29:23.840250 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.840176 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:23.845209 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.845188 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:23.984505 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:23.984445 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:24.028138 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028105 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bb2q\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-kube-api-access-4bb2q\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.028389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028163 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-ca-configmap\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.028389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-kubeconfig\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.028389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028244 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-token\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.028389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028288 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/792f40ca-011a-4c34-bf8b-830f0b5f230c-local-certs\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.028389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028323 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-cacerts\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.028389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.028346 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-dns-cert\") pod \"792f40ca-011a-4c34-bf8b-830f0b5f230c\" (UID: \"792f40ca-011a-4c34-bf8b-830f0b5f230c\") " Jul 06 22:29:24.029125 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.029093 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:29:24.031035 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.030995 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-cacerts" (OuterVolumeSpecName: "cacerts") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:29:24.031360 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.031329 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:29:24.031481 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.031357 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-kube-api-access-4bb2q" (OuterVolumeSpecName: "kube-api-access-4bb2q") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "kube-api-access-4bb2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:29:24.031481 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.031330 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-token" (OuterVolumeSpecName: "istio-token") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:29:24.031589 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.031567 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:29:24.032481 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.032412 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/792f40ca-011a-4c34-bf8b-830f0b5f230c-local-certs" (OuterVolumeSpecName: "local-certs") pod "792f40ca-011a-4c34-bf8b-830f0b5f230c" (UID: "792f40ca-011a-4c34-bf8b-830f0b5f230c"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:29:24.129203 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129158 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-kubeconfig\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.129203 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129200 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-token\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.129414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129215 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/792f40ca-011a-4c34-bf8b-830f0b5f230c-local-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.129414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129224 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-cacerts\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.129414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129234 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-dns-cert\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.129414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129243 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bb2q\" (UniqueName: \"kubernetes.io/projected/792f40ca-011a-4c34-bf8b-830f0b5f230c-kube-api-access-4bb2q\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.129414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.129252 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/792f40ca-011a-4c34-bf8b-830f0b5f230c-istio-csr-ca-configmap\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:29:24.318259 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.318174 2564 generic.go:356] "Generic (PLEG): container finished" podID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerID="3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f" exitCode=0 Jul 06 22:29:24.323754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.318491 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" event={"ID":"792f40ca-011a-4c34-bf8b-830f0b5f230c","Type":"ContainerDied","Data":"3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f"} Jul 06 22:29:24.323754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.318569 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" event={"ID":"792f40ca-011a-4c34-bf8b-830f0b5f230c","Type":"ContainerDied","Data":"ab7d93a881132291c55a51ec804a3cf26e30ec0c883160e7d8443d2869a6a86c"} Jul 06 22:29:24.323754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.318588 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-278f7" Jul 06 22:29:24.323754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.318615 2564 scope.go:117] "RemoveContainer" containerID="3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f" Jul 06 22:29:24.324398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.324375 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:24.325768 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.325748 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4vkz5j" Jul 06 22:29:24.333746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.333721 2564 scope.go:117] "RemoveContainer" containerID="3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f" Jul 06 22:29:24.334083 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:29:24.334060 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f\": container with ID starting with 3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f not found: ID does not exist" containerID="3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f" Jul 06 22:29:24.334155 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.334090 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f"} err="failed to get container status \"3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f\": rpc error: code = NotFound desc = could not find container \"3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f\": container with ID starting with 3054a24396a8e7693588e8192653686127447ead70c5991ebf6c100f461bb19f not found: ID does not exist" Jul 06 22:29:24.372790 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.372753 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-278f7"] Jul 06 22:29:24.393450 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.386754 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-278f7"] Jul 06 22:29:24.809106 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:24.809073 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" path="/var/lib/kubelet/pods/792f40ca-011a-4c34-bf8b-830f0b5f230c/volumes" Jul 06 22:29:26.330372 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:26.330331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" event={"ID":"4e0a57f9-b003-45f9-86a9-b0752d1aede5","Type":"ContainerStarted","Data":"3477da4a045f14ca8f12c4fc9b2d5661d25b09dde0114c6b641c0d950ce8db16"} Jul 06 22:29:26.330806 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:26.330400 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:29:26.349157 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:26.349110 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" podStartSLOduration=2.64670708 podStartE2EDuration="5.349093943s" totalImagesPullingTime="2.702386863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:29:26.346686427 +0000 UTC m=+328.083088415" watchObservedRunningTime="2026-07-06 22:29:26.349093943 +0000 UTC m=+328.085495930" Jul 06 22:29:28.286195 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:28.286163 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-2dkjj" Jul 06 22:29:28.288189 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:28.288164 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6c44h" Jul 06 22:29:57.340452 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:29:57.340349 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xthph" Jul 06 22:30:10.822911 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.822876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:10.968473 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.968420 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2smt4"] Jul 06 22:30:10.968807 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.968790 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerName="discovery" Jul 06 22:30:10.968886 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.968808 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerName="discovery" Jul 06 22:30:10.968938 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.968912 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="792f40ca-011a-4c34-bf8b-830f0b5f230c" containerName="discovery" Jul 06 22:30:10.971538 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.971517 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:10.974197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.974172 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-p66mb\"" Jul 06 22:30:10.974338 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.974322 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 22:30:10.976053 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:10.976038 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 22:30:11.005307 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.005276 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2smt4"] Jul 06 22:30:11.017827 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.017795 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:11.113795 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.113763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:11.126051 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.126023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgp5t\" (UniqueName: \"kubernetes.io/projected/7879d7b8-ff6a-4c5e-91c2-15036765e115-kube-api-access-zgp5t\") pod \"kuadrant-operator-catalog-2smt4\" (UID: \"7879d7b8-ff6a-4c5e-91c2-15036765e115\") " pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:11.227166 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.227128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgp5t\" (UniqueName: \"kubernetes.io/projected/7879d7b8-ff6a-4c5e-91c2-15036765e115-kube-api-access-zgp5t\") pod \"kuadrant-operator-catalog-2smt4\" (UID: \"7879d7b8-ff6a-4c5e-91c2-15036765e115\") " pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:11.236284 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.236246 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgp5t\" (UniqueName: \"kubernetes.io/projected/7879d7b8-ff6a-4c5e-91c2-15036765e115-kube-api-access-zgp5t\") pod \"kuadrant-operator-catalog-2smt4\" (UID: \"7879d7b8-ff6a-4c5e-91c2-15036765e115\") " pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:11.258959 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.258924 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2smt4"] Jul 06 22:30:11.259303 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.259287 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:11.397059 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.396966 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2smt4"] Jul 06 22:30:11.399237 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:30:11.399212 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7879d7b8_ff6a_4c5e_91c2_15036765e115.slice/crio-885a9935244f0aa2117aeaed537de3a5a2edb2b2b6e5e9fcac4b41b82a83c40d WatchSource:0}: Error finding container 885a9935244f0aa2117aeaed537de3a5a2edb2b2b6e5e9fcac4b41b82a83c40d: Status 404 returned error can't find the container with id 885a9935244f0aa2117aeaed537de3a5a2edb2b2b6e5e9fcac4b41b82a83c40d Jul 06 22:30:11.493042 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.493003 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zwcc4"] Jul 06 22:30:11.495793 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.495770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:11.514872 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.514842 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zwcc4"] Jul 06 22:30:11.523243 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.523218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" event={"ID":"7879d7b8-ff6a-4c5e-91c2-15036765e115","Type":"ContainerStarted","Data":"885a9935244f0aa2117aeaed537de3a5a2edb2b2b6e5e9fcac4b41b82a83c40d"} Jul 06 22:30:11.629441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.629402 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pkbg\" (UniqueName: \"kubernetes.io/projected/d4fa971d-7b06-4efc-9427-a69e25594da7-kube-api-access-9pkbg\") pod \"kuadrant-operator-catalog-zwcc4\" (UID: \"d4fa971d-7b06-4efc-9427-a69e25594da7\") " pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:11.717198 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.717113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:11.730349 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.730324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pkbg\" (UniqueName: \"kubernetes.io/projected/d4fa971d-7b06-4efc-9427-a69e25594da7-kube-api-access-9pkbg\") pod \"kuadrant-operator-catalog-zwcc4\" (UID: \"d4fa971d-7b06-4efc-9427-a69e25594da7\") " pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:11.740313 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.740289 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pkbg\" (UniqueName: \"kubernetes.io/projected/d4fa971d-7b06-4efc-9427-a69e25594da7-kube-api-access-9pkbg\") pod \"kuadrant-operator-catalog-zwcc4\" (UID: \"d4fa971d-7b06-4efc-9427-a69e25594da7\") " pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:11.804813 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.804782 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:11.896522 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.896419 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:11.947807 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:11.947700 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zwcc4"] Jul 06 22:30:11.950302 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:30:11.950278 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fa971d_7b06_4efc_9427_a69e25594da7.slice/crio-684386ff945900964927fafd88fcdbe34a48d323fd09f3e21279c45c53b676d8 WatchSource:0}: Error finding container 684386ff945900964927fafd88fcdbe34a48d323fd09f3e21279c45c53b676d8: Status 404 returned error can't find the container with id 684386ff945900964927fafd88fcdbe34a48d323fd09f3e21279c45c53b676d8 Jul 06 22:30:12.534258 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:12.534219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" event={"ID":"d4fa971d-7b06-4efc-9427-a69e25594da7","Type":"ContainerStarted","Data":"684386ff945900964927fafd88fcdbe34a48d323fd09f3e21279c45c53b676d8"} Jul 06 22:30:13.539704 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.539584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" event={"ID":"7879d7b8-ff6a-4c5e-91c2-15036765e115","Type":"ContainerStarted","Data":"96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4"} Jul 06 22:30:13.539704 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.539648 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" podUID="7879d7b8-ff6a-4c5e-91c2-15036765e115" containerName="registry-server" containerID="cri-o://96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4" gracePeriod=2 Jul 06 22:30:13.541143 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.541115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" event={"ID":"d4fa971d-7b06-4efc-9427-a69e25594da7","Type":"ContainerStarted","Data":"67409ca122a6e9feaa7d9f6d4cbb74824e49261f98abed6df29d3a36a5d4fac3"} Jul 06 22:30:13.617912 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.617847 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" podStartSLOduration=1.765588132 podStartE2EDuration="3.617830929s" totalImagesPullingTime="1.852242797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:13.579930878 +0000 UTC m=+375.316332865" watchObservedRunningTime="2026-07-06 22:30:13.617830929 +0000 UTC m=+375.354232915" Jul 06 22:30:13.618217 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.618197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" podStartSLOduration=1.316099431 podStartE2EDuration="2.61819112s" totalImagesPullingTime="1.302091689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:13.616325896 +0000 UTC m=+375.352727883" watchObservedRunningTime="2026-07-06 22:30:13.61819112 +0000 UTC m=+375.354593107" Jul 06 22:30:13.786201 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.786170 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:13.949290 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.949253 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgp5t\" (UniqueName: \"kubernetes.io/projected/7879d7b8-ff6a-4c5e-91c2-15036765e115-kube-api-access-zgp5t\") pod \"7879d7b8-ff6a-4c5e-91c2-15036765e115\" (UID: \"7879d7b8-ff6a-4c5e-91c2-15036765e115\") " Jul 06 22:30:13.951499 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:13.951458 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7879d7b8-ff6a-4c5e-91c2-15036765e115-kube-api-access-zgp5t" (OuterVolumeSpecName: "kube-api-access-zgp5t") pod "7879d7b8-ff6a-4c5e-91c2-15036765e115" (UID: "7879d7b8-ff6a-4c5e-91c2-15036765e115"). InnerVolumeSpecName "kube-api-access-zgp5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:30:14.049934 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.049893 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgp5t\" (UniqueName: \"kubernetes.io/projected/7879d7b8-ff6a-4c5e-91c2-15036765e115-kube-api-access-zgp5t\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:30:14.546326 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.546241 2564 generic.go:356] "Generic (PLEG): container finished" podID="7879d7b8-ff6a-4c5e-91c2-15036765e115" containerID="96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4" exitCode=0 Jul 06 22:30:14.546326 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.546282 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" Jul 06 22:30:14.546796 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.546331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" event={"ID":"7879d7b8-ff6a-4c5e-91c2-15036765e115","Type":"ContainerDied","Data":"96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4"} Jul 06 22:30:14.546796 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.546381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2smt4" event={"ID":"7879d7b8-ff6a-4c5e-91c2-15036765e115","Type":"ContainerDied","Data":"885a9935244f0aa2117aeaed537de3a5a2edb2b2b6e5e9fcac4b41b82a83c40d"} Jul 06 22:30:14.546796 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.546401 2564 scope.go:117] "RemoveContainer" containerID="96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4" Jul 06 22:30:14.555626 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.555604 2564 scope.go:117] "RemoveContainer" containerID="96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4" Jul 06 22:30:14.555889 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:30:14.555871 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4\": container with ID starting with 96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4 not found: ID does not exist" containerID="96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4" Jul 06 22:30:14.555935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.555900 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4"} err="failed to get container status \"96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4\": rpc error: code = NotFound desc = could not find container \"96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4\": container with ID starting with 96d365b6fc4d2ded04880158542e7c3c37e56e9d564e821865cc50a6c7e084a4 not found: ID does not exist" Jul 06 22:30:14.577550 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.577515 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2smt4"] Jul 06 22:30:14.582305 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.582278 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2smt4"] Jul 06 22:30:14.809200 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:14.809125 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7879d7b8-ff6a-4c5e-91c2-15036765e115" path="/var/lib/kubelet/pods/7879d7b8-ff6a-4c5e-91c2-15036765e115/volumes" Jul 06 22:30:21.805123 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:21.805089 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:21.805750 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:21.805147 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:21.828577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:21.828548 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:22.601848 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:22.601817 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zwcc4" Jul 06 22:30:39.819340 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:39.819301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:43.374160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:43.374128 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:44.525132 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.525096 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g"] Jul 06 22:30:44.525542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.525416 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7879d7b8-ff6a-4c5e-91c2-15036765e115" containerName="registry-server" Jul 06 22:30:44.525542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.525442 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7879d7b8-ff6a-4c5e-91c2-15036765e115" containerName="registry-server" Jul 06 22:30:44.525542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.525507 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7879d7b8-ff6a-4c5e-91c2-15036765e115" containerName="registry-server" Jul 06 22:30:44.528485 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.528466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.531355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.531333 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8cgv8\"" Jul 06 22:30:44.544936 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.544910 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g"] Jul 06 22:30:44.610490 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.610443 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2d89a57-d18d-48c4-9c87-299b8d953334-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.610673 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.610550 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td46v\" (UniqueName: \"kubernetes.io/projected/d2d89a57-d18d-48c4-9c87-299b8d953334-kube-api-access-td46v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.711709 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.711672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2d89a57-d18d-48c4-9c87-299b8d953334-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.711885 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.711744 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td46v\" (UniqueName: \"kubernetes.io/projected/d2d89a57-d18d-48c4-9c87-299b8d953334-kube-api-access-td46v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.712093 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.712067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2d89a57-d18d-48c4-9c87-299b8d953334-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.720706 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.720681 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td46v\" (UniqueName: \"kubernetes.io/projected/d2d89a57-d18d-48c4-9c87-299b8d953334-kube-api-access-td46v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.838731 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.838699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:44.976806 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:44.976780 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g"] Jul 06 22:30:44.979674 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:30:44.979638 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d89a57_d18d_48c4_9c87_299b8d953334.slice/crio-e576a33db5596c7ca2c4f4db2993395ebf83d643b9f89de6efe8a386ce4355e8 WatchSource:0}: Error finding container e576a33db5596c7ca2c4f4db2993395ebf83d643b9f89de6efe8a386ce4355e8: Status 404 returned error can't find the container with id e576a33db5596c7ca2c4f4db2993395ebf83d643b9f89de6efe8a386ce4355e8 Jul 06 22:30:45.237741 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:45.237657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:45.676008 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:45.675972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" event={"ID":"d2d89a57-d18d-48c4-9c87-299b8d953334","Type":"ContainerStarted","Data":"e576a33db5596c7ca2c4f4db2993395ebf83d643b9f89de6efe8a386ce4355e8"} Jul 06 22:30:46.534316 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.534281 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9"] Jul 06 22:30:46.538072 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.538046 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:30:46.540516 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.540492 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tjdr7\"" Jul 06 22:30:46.546554 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.546530 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9"] Jul 06 22:30:46.629664 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.629615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd9vs\" (UniqueName: \"kubernetes.io/projected/777a43f2-e289-4695-8d44-eb15010c776e-kube-api-access-pd9vs\") pod \"limitador-operator-controller-manager-859d7b78bd-htpq9\" (UID: \"777a43f2-e289-4695-8d44-eb15010c776e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:30:46.730204 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.730171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9vs\" (UniqueName: \"kubernetes.io/projected/777a43f2-e289-4695-8d44-eb15010c776e-kube-api-access-pd9vs\") pod \"limitador-operator-controller-manager-859d7b78bd-htpq9\" (UID: \"777a43f2-e289-4695-8d44-eb15010c776e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:30:46.740821 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.740797 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9vs\" (UniqueName: \"kubernetes.io/projected/777a43f2-e289-4695-8d44-eb15010c776e-kube-api-access-pd9vs\") pod \"limitador-operator-controller-manager-859d7b78bd-htpq9\" (UID: \"777a43f2-e289-4695-8d44-eb15010c776e\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:30:46.851484 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.851454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:30:46.992212 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:46.992127 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9"] Jul 06 22:30:46.996984 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:30:46.996952 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777a43f2_e289_4695_8d44_eb15010c776e.slice/crio-7291a50c58eef7fb582c2d62c39dd2821b21aef8e7e6281052ba5f15f245c74b WatchSource:0}: Error finding container 7291a50c58eef7fb582c2d62c39dd2821b21aef8e7e6281052ba5f15f245c74b: Status 404 returned error can't find the container with id 7291a50c58eef7fb582c2d62c39dd2821b21aef8e7e6281052ba5f15f245c74b Jul 06 22:30:47.087859 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:47.087823 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:47.193201 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:47.193106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:47.286440 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:47.286391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:47.429261 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:47.429228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:47.687742 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:47.687698 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" event={"ID":"777a43f2-e289-4695-8d44-eb15010c776e","Type":"ContainerStarted","Data":"7291a50c58eef7fb582c2d62c39dd2821b21aef8e7e6281052ba5f15f245c74b"} Jul 06 22:30:48.914628 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:48.914586 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:49.162003 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.161966 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd"] Jul 06 22:30:49.166024 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.165947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:30:49.169935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.169905 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5gkv5\"" Jul 06 22:30:49.170076 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.169985 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 22:30:49.175198 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.175139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd"] Jul 06 22:30:49.252085 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.252044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjdh\" (UniqueName: \"kubernetes.io/projected/49b18a4b-4dd0-464d-b8df-49a3448248fc-kube-api-access-2mjdh\") pod \"dns-operator-controller-manager-864dd78c4c-7vxgd\" (UID: \"49b18a4b-4dd0-464d-b8df-49a3448248fc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:30:49.352784 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.352745 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjdh\" (UniqueName: \"kubernetes.io/projected/49b18a4b-4dd0-464d-b8df-49a3448248fc-kube-api-access-2mjdh\") pod \"dns-operator-controller-manager-864dd78c4c-7vxgd\" (UID: \"49b18a4b-4dd0-464d-b8df-49a3448248fc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:30:49.370602 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.370567 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjdh\" (UniqueName: \"kubernetes.io/projected/49b18a4b-4dd0-464d-b8df-49a3448248fc-kube-api-access-2mjdh\") pod \"dns-operator-controller-manager-864dd78c4c-7vxgd\" (UID: \"49b18a4b-4dd0-464d-b8df-49a3448248fc\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:30:49.481489 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.481400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:30:49.599131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.599089 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:49.839108 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.839078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:49.946261 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:49.946226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd"] Jul 06 22:30:49.949132 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:30:49.949108 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b18a4b_4dd0_464d_b8df_49a3448248fc.slice/crio-77c036c5879ec276ba755f1892cc4c56ef45c70a9d37cf3bb3679997594b9ef0 WatchSource:0}: Error finding container 77c036c5879ec276ba755f1892cc4c56ef45c70a9d37cf3bb3679997594b9ef0: Status 404 returned error can't find the container with id 77c036c5879ec276ba755f1892cc4c56ef45c70a9d37cf3bb3679997594b9ef0 Jul 06 22:30:50.021068 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.021036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:50.153176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.153143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:50.283110 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.283074 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:50.708419 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.708384 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" event={"ID":"777a43f2-e289-4695-8d44-eb15010c776e","Type":"ContainerStarted","Data":"a1d90686f0e0671a12988c653ed54f15374142d20e50ffd18073d8e3632d3162"} Jul 06 22:30:50.708618 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.708467 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:30:50.712683 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.712652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" event={"ID":"d2d89a57-d18d-48c4-9c87-299b8d953334","Type":"ContainerStarted","Data":"f59cf17539f587653be902da1b0eb2db2ca7f3024800807d49a9a97e2f435bdc"} Jul 06 22:30:50.712818 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.712723 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:30:50.714270 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.714246 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" event={"ID":"49b18a4b-4dd0-464d-b8df-49a3448248fc","Type":"ContainerStarted","Data":"77c036c5879ec276ba755f1892cc4c56ef45c70a9d37cf3bb3679997594b9ef0"} Jul 06 22:30:50.727147 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.727099 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" podStartSLOduration=1.920104722 podStartE2EDuration="4.727082776s" totalImagesPullingTime="2.806978054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:50.726519817 +0000 UTC m=+412.462921805" watchObservedRunningTime="2026-07-06 22:30:50.727082776 +0000 UTC m=+412.463484741" Jul 06 22:30:50.752001 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:50.751946 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" podStartSLOduration=1.87021864 podStartE2EDuration="6.751926546s" totalImagesPullingTime="4.881707906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:50.749727712 +0000 UTC m=+412.486129698" watchObservedRunningTime="2026-07-06 22:30:50.751926546 +0000 UTC m=+412.488328580" Jul 06 22:30:52.270891 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.270802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:52.388501 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.388461 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:52.608378 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.608348 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w8f97"] Jul 06 22:30:52.612057 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.612032 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:30:52.614721 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.614700 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2wps4\"" Jul 06 22:30:52.624003 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.623973 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w8f97"] Jul 06 22:30:52.689160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.689118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5zz\" (UniqueName: \"kubernetes.io/projected/af9bacca-4e0f-4dab-8bf9-f55af61cee27-kube-api-access-jc5zz\") pod \"authorino-operator-75cc8d5cb7-w8f97\" (UID: \"af9bacca-4e0f-4dab-8bf9-f55af61cee27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:30:52.726176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.726132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" event={"ID":"49b18a4b-4dd0-464d-b8df-49a3448248fc","Type":"ContainerStarted","Data":"eef61da3b801aabb3ffbcd5c071bd2810d2e9346b45b264c800950ada5966a64"} Jul 06 22:30:52.726358 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.726211 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:30:52.746366 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.746313 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" podStartSLOduration=1.905880722 podStartE2EDuration="3.746294261s" totalImagesPullingTime="1.840413539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:52.745869686 +0000 UTC m=+414.482271676" watchObservedRunningTime="2026-07-06 22:30:52.746294261 +0000 UTC m=+414.482696247" Jul 06 22:30:52.790107 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.790072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5zz\" (UniqueName: \"kubernetes.io/projected/af9bacca-4e0f-4dab-8bf9-f55af61cee27-kube-api-access-jc5zz\") pod \"authorino-operator-75cc8d5cb7-w8f97\" (UID: \"af9bacca-4e0f-4dab-8bf9-f55af61cee27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:30:52.801347 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.801317 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5zz\" (UniqueName: \"kubernetes.io/projected/af9bacca-4e0f-4dab-8bf9-f55af61cee27-kube-api-access-jc5zz\") pod \"authorino-operator-75cc8d5cb7-w8f97\" (UID: \"af9bacca-4e0f-4dab-8bf9-f55af61cee27\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:30:52.848214 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.848183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:52.922678 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:52.922588 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:30:53.023331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:53.023269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:53.071014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:53.070983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-w8f97"] Jul 06 22:30:53.074140 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:30:53.074111 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf9bacca_4e0f_4dab_8bf9_f55af61cee27.slice/crio-ee4cfc16de913b597b77e846ea0ba41d8be450bab4eec04911771d5b7a4cf4f2 WatchSource:0}: Error finding container ee4cfc16de913b597b77e846ea0ba41d8be450bab4eec04911771d5b7a4cf4f2: Status 404 returned error can't find the container with id ee4cfc16de913b597b77e846ea0ba41d8be450bab4eec04911771d5b7a4cf4f2 Jul 06 22:30:53.733454 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:53.733399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" event={"ID":"af9bacca-4e0f-4dab-8bf9-f55af61cee27","Type":"ContainerStarted","Data":"ee4cfc16de913b597b77e846ea0ba41d8be450bab4eec04911771d5b7a4cf4f2"} Jul 06 22:30:54.814746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:54.814699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:54.886089 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:54.886054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:55.746572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:55.746534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" event={"ID":"af9bacca-4e0f-4dab-8bf9-f55af61cee27","Type":"ContainerStarted","Data":"ddda5048203d9f03fd880419ecf3f667f6181f941637337d19fd78b650c9fb7a"} Jul 06 22:30:55.746775 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:55.746663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:30:55.768512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:55.768455 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" podStartSLOduration=1.537726901 podStartE2EDuration="3.76843844s" totalImagesPullingTime="2.230711539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:30:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:30:55.767226488 +0000 UTC m=+417.503628474" watchObservedRunningTime="2026-07-06 22:30:55.76843844 +0000 UTC m=+417.504840417" Jul 06 22:30:59.641935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:59.641897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:30:59.710771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:30:59.710738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:01.420981 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:01.420944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:01.521135 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:01.521101 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:01.621050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:01.621017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:01.722232 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:01.722141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:01.722379 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:01.722314 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.269184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.269153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:03.363811 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.363772 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:03.504918 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.504887 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g"] Jul 06 22:31:03.505129 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.505100 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" containerName="manager" containerID="cri-o://f59cf17539f587653be902da1b0eb2db2ca7f3024800807d49a9a97e2f435bdc" gracePeriod=2 Jul 06 22:31:03.506553 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.506531 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g"] Jul 06 22:31:03.516863 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.516829 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.530182 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.530152 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9"] Jul 06 22:31:03.530462 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.530417 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" podUID="777a43f2-e289-4695-8d44-eb15010c776e" containerName="manager" containerID="cri-o://a1d90686f0e0671a12988c653ed54f15374142d20e50ffd18073d8e3632d3162" gracePeriod=2 Jul 06 22:31:03.538793 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.538756 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.540150 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.540122 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65"] Jul 06 22:31:03.540573 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.540553 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" containerName="manager" Jul 06 22:31:03.540573 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.540573 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" containerName="manager" Jul 06 22:31:03.540716 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.540643 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" containerName="manager" Jul 06 22:31:03.543507 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.543487 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.549751 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.549713 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.551546 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.551523 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9"] Jul 06 22:31:03.552085 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.552055 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-htpq9\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="777a43f2-e289-4695-8d44-eb15010c776e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:03.570006 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.569980 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b"] Jul 06 22:31:03.570350 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.570334 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="777a43f2-e289-4695-8d44-eb15010c776e" containerName="manager" Jul 06 22:31:03.570442 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.570353 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="777a43f2-e289-4695-8d44-eb15010c776e" containerName="manager" Jul 06 22:31:03.570511 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.570499 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="777a43f2-e289-4695-8d44-eb15010c776e" containerName="manager" Jul 06 22:31:03.573454 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.573418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:03.586658 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.586623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vgbm\" (UniqueName: \"kubernetes.io/projected/39f966d8-5e63-484d-a7f1-1621a3ad0b69-kube-api-access-2vgbm\") pod \"limitador-operator-controller-manager-859d7b78bd-rm27b\" (UID: \"39f966d8-5e63-484d-a7f1-1621a3ad0b69\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:03.586812 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.586689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgn4v\" (UniqueName: \"kubernetes.io/projected/28d4f961-2402-4004-89af-6f24838dfcc8-kube-api-access-lgn4v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hkk65\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.586812 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.586753 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d4f961-2402-4004-89af-6f24838dfcc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hkk65\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.587792 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.587770 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65"] Jul 06 22:31:03.599073 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.599040 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b"] Jul 06 22:31:03.633699 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.633651 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.635786 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.635757 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-htpq9\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="777a43f2-e289-4695-8d44-eb15010c776e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:03.687409 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.687366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgn4v\" (UniqueName: \"kubernetes.io/projected/28d4f961-2402-4004-89af-6f24838dfcc8-kube-api-access-lgn4v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hkk65\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.687629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.687417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d4f961-2402-4004-89af-6f24838dfcc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hkk65\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.687629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.687555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vgbm\" (UniqueName: \"kubernetes.io/projected/39f966d8-5e63-484d-a7f1-1621a3ad0b69-kube-api-access-2vgbm\") pod \"limitador-operator-controller-manager-859d7b78bd-rm27b\" (UID: \"39f966d8-5e63-484d-a7f1-1621a3ad0b69\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:03.687963 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.687940 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d4f961-2402-4004-89af-6f24838dfcc8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hkk65\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.696477 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.696215 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vgbm\" (UniqueName: \"kubernetes.io/projected/39f966d8-5e63-484d-a7f1-1621a3ad0b69-kube-api-access-2vgbm\") pod \"limitador-operator-controller-manager-859d7b78bd-rm27b\" (UID: \"39f966d8-5e63-484d-a7f1-1621a3ad0b69\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:03.696627 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.696560 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgn4v\" (UniqueName: \"kubernetes.io/projected/28d4f961-2402-4004-89af-6f24838dfcc8-kube-api-access-lgn4v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hkk65\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.741628 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.740721 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-7vxgd" Jul 06 22:31:03.743814 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.743636 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.749848 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.749665 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-htpq9\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="777a43f2-e289-4695-8d44-eb15010c776e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:03.789700 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.789665 2564 generic.go:356] "Generic (PLEG): container finished" podID="777a43f2-e289-4695-8d44-eb15010c776e" containerID="a1d90686f0e0671a12988c653ed54f15374142d20e50ffd18073d8e3632d3162" exitCode=0 Jul 06 22:31:03.791391 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.791362 2564 generic.go:356] "Generic (PLEG): container finished" podID="d2d89a57-d18d-48c4-9c87-299b8d953334" containerID="f59cf17539f587653be902da1b0eb2db2ca7f3024800807d49a9a97e2f435bdc" exitCode=0 Jul 06 22:31:03.791547 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.791502 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e576a33db5596c7ca2c4f4db2993395ebf83d643b9f89de6efe8a386ce4355e8" Jul 06 22:31:03.799535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.799498 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.801809 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.801780 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.803120 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.803100 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:03.804329 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.804300 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-htpq9\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="777a43f2-e289-4695-8d44-eb15010c776e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:03.806614 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.806584 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:03.808593 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:03.808570 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-htpq9\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="777a43f2-e289-4695-8d44-eb15010c776e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:03.955823 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.955778 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:03.962577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.962549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:03.989835 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.989799 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td46v\" (UniqueName: \"kubernetes.io/projected/d2d89a57-d18d-48c4-9c87-299b8d953334-kube-api-access-td46v\") pod \"d2d89a57-d18d-48c4-9c87-299b8d953334\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " Jul 06 22:31:03.990014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.989895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2d89a57-d18d-48c4-9c87-299b8d953334-extensions-socket-volume\") pod \"d2d89a57-d18d-48c4-9c87-299b8d953334\" (UID: \"d2d89a57-d18d-48c4-9c87-299b8d953334\") " Jul 06 22:31:03.990014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.989947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd9vs\" (UniqueName: \"kubernetes.io/projected/777a43f2-e289-4695-8d44-eb15010c776e-kube-api-access-pd9vs\") pod \"777a43f2-e289-4695-8d44-eb15010c776e\" (UID: \"777a43f2-e289-4695-8d44-eb15010c776e\") " Jul 06 22:31:03.990459 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.990406 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2d89a57-d18d-48c4-9c87-299b8d953334-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d2d89a57-d18d-48c4-9c87-299b8d953334" (UID: "d2d89a57-d18d-48c4-9c87-299b8d953334"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:31:03.992026 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.992000 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a43f2-e289-4695-8d44-eb15010c776e-kube-api-access-pd9vs" (OuterVolumeSpecName: "kube-api-access-pd9vs") pod "777a43f2-e289-4695-8d44-eb15010c776e" (UID: "777a43f2-e289-4695-8d44-eb15010c776e"). InnerVolumeSpecName "kube-api-access-pd9vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:31:03.992121 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:03.992045 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d89a57-d18d-48c4-9c87-299b8d953334-kube-api-access-td46v" (OuterVolumeSpecName: "kube-api-access-td46v") pod "d2d89a57-d18d-48c4-9c87-299b8d953334" (UID: "d2d89a57-d18d-48c4-9c87-299b8d953334"). InnerVolumeSpecName "kube-api-access-td46v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:31:04.091213 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.091161 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td46v\" (UniqueName: \"kubernetes.io/projected/d2d89a57-d18d-48c4-9c87-299b8d953334-kube-api-access-td46v\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:31:04.091213 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.091212 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d2d89a57-d18d-48c4-9c87-299b8d953334-extensions-socket-volume\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:31:04.091456 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.091228 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd9vs\" (UniqueName: \"kubernetes.io/projected/777a43f2-e289-4695-8d44-eb15010c776e-kube-api-access-pd9vs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:31:04.109028 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.108991 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65"] Jul 06 22:31:04.113740 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:31:04.113711 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28d4f961_2402_4004_89af_6f24838dfcc8.slice/crio-f4d42ed5a140c69b41e8dd0ff5766dc23c82e0e36e2c22f21c2a384957ad2619 WatchSource:0}: Error finding container f4d42ed5a140c69b41e8dd0ff5766dc23c82e0e36e2c22f21c2a384957ad2619: Status 404 returned error can't find the container with id f4d42ed5a140c69b41e8dd0ff5766dc23c82e0e36e2c22f21c2a384957ad2619 Jul 06 22:31:04.125290 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.125269 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b"] Jul 06 22:31:04.128209 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:31:04.128179 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f966d8_5e63_484d_a7f1_1621a3ad0b69.slice/crio-59a69eb9b8af7f6e4e73c71cd01c9c0c264233f0a971f19e9720b5588ecc1036 WatchSource:0}: Error finding container 59a69eb9b8af7f6e4e73c71cd01c9c0c264233f0a971f19e9720b5588ecc1036: Status 404 returned error can't find the container with id 59a69eb9b8af7f6e4e73c71cd01c9c0c264233f0a971f19e9720b5588ecc1036 Jul 06 22:31:04.797948 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.797863 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" event={"ID":"28d4f961-2402-4004-89af-6f24838dfcc8","Type":"ContainerStarted","Data":"c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d"} Jul 06 22:31:04.797948 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.797919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" event={"ID":"28d4f961-2402-4004-89af-6f24838dfcc8","Type":"ContainerStarted","Data":"f4d42ed5a140c69b41e8dd0ff5766dc23c82e0e36e2c22f21c2a384957ad2619"} Jul 06 22:31:04.798377 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.797957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:04.799286 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.799268 2564 scope.go:117] "RemoveContainer" containerID="a1d90686f0e0671a12988c653ed54f15374142d20e50ffd18073d8e3632d3162" Jul 06 22:31:04.799368 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.799285 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:04.800111 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:04.800069 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bb56g\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:04.801032 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.801010 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bb56g" Jul 06 22:31:04.801129 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.801036 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" event={"ID":"39f966d8-5e63-484d-a7f1-1621a3ad0b69","Type":"ContainerStarted","Data":"519d37af2a2d9af470e2e2c930189b82ef76049b535f7731c5a965a62b985187"} Jul 06 22:31:04.801129 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.801074 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" event={"ID":"39f966d8-5e63-484d-a7f1-1621a3ad0b69","Type":"ContainerStarted","Data":"59a69eb9b8af7f6e4e73c71cd01c9c0c264233f0a971f19e9720b5588ecc1036"} Jul 06 22:31:04.802049 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:04.802016 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-htpq9\" is forbidden: User \"system:node:ip-10-0-135-55.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-55.ec2.internal' and this object" podUID="777a43f2-e289-4695-8d44-eb15010c776e" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-htpq9" Jul 06 22:31:04.811158 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.811135 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="777a43f2-e289-4695-8d44-eb15010c776e" path="/var/lib/kubelet/pods/777a43f2-e289-4695-8d44-eb15010c776e/volumes" Jul 06 22:31:04.811576 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.811559 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2d89a57-d18d-48c4-9c87-299b8d953334" path="/var/lib/kubelet/pods/d2d89a57-d18d-48c4-9c87-299b8d953334/volumes" Jul 06 22:31:04.848620 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.848575 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" podStartSLOduration=1.848559754 podStartE2EDuration="1.848559754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:04.824998789 +0000 UTC m=+426.561400779" watchObservedRunningTime="2026-07-06 22:31:04.848559754 +0000 UTC m=+426.584961741" Jul 06 22:31:04.848744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.848653 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" podStartSLOduration=1.848649364 podStartE2EDuration="1.848649364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:04.847682113 +0000 UTC m=+426.584084100" watchObservedRunningTime="2026-07-06 22:31:04.848649364 +0000 UTC m=+426.585051429" Jul 06 22:31:04.914552 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.914521 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68"] Jul 06 22:31:04.918916 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.918896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:04.929471 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.929451 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68"] Jul 06 22:31:04.999076 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.999036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5vw\" (UniqueName: \"kubernetes.io/projected/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-kube-api-access-rx5vw\") pod \"kuadrant-operator-controller-manager-86878c769f-gnd68\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:04.999282 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:04.999081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gnd68\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.100537 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.100503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5vw\" (UniqueName: \"kubernetes.io/projected/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-kube-api-access-rx5vw\") pod \"kuadrant-operator-controller-manager-86878c769f-gnd68\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.100537 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.100544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gnd68\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.100898 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.100883 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-gnd68\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.116098 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.116074 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5vw\" (UniqueName: \"kubernetes.io/projected/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-kube-api-access-rx5vw\") pod \"kuadrant-operator-controller-manager-86878c769f-gnd68\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.229323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.229288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.365552 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:31:05.365524 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4971d3fe_bbfd_4ef3_a86e_f4a2e0042f15.slice/crio-63d413b4ce250de06464ee1b50f01e3114cafac4d176c5678d16b74c5daeaa96 WatchSource:0}: Error finding container 63d413b4ce250de06464ee1b50f01e3114cafac4d176c5678d16b74c5daeaa96: Status 404 returned error can't find the container with id 63d413b4ce250de06464ee1b50f01e3114cafac4d176c5678d16b74c5daeaa96 Jul 06 22:31:05.365763 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.365738 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68"] Jul 06 22:31:05.711215 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.711132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:05.808839 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.808800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" event={"ID":"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15","Type":"ContainerStarted","Data":"05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a"} Jul 06 22:31:05.809260 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.809115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:05.809260 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.809151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" event={"ID":"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15","Type":"ContainerStarted","Data":"63d413b4ce250de06464ee1b50f01e3114cafac4d176c5678d16b74c5daeaa96"} Jul 06 22:31:05.809260 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.809230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:05.820347 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.820321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:05.828513 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:05.828471 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" podStartSLOduration=1.828457535 podStartE2EDuration="1.828457535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:05.827714276 +0000 UTC m=+427.564116262" watchObservedRunningTime="2026-07-06 22:31:05.828457535 +0000 UTC m=+427.564859516" Jul 06 22:31:06.754562 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:06.754522 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-w8f97" Jul 06 22:31:12.273489 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:12.273434 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:12.380726 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:12.380689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:15.811316 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:15.811287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:16.547176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:16.547137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:16.655961 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:16.655930 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:16.815922 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:16.815837 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:31:16.816309 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:16.815923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rm27b" Jul 06 22:31:16.886005 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:16.885970 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65"] Jul 06 22:31:16.886184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:16.886159 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" podUID="28d4f961-2402-4004-89af-6f24838dfcc8" containerName="manager" containerID="cri-o://c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d" gracePeriod=10 Jul 06 22:31:17.166301 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.166274 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:17.196285 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.196261 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d4f961-2402-4004-89af-6f24838dfcc8-extensions-socket-volume\") pod \"28d4f961-2402-4004-89af-6f24838dfcc8\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " Jul 06 22:31:17.196460 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.196297 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgn4v\" (UniqueName: \"kubernetes.io/projected/28d4f961-2402-4004-89af-6f24838dfcc8-kube-api-access-lgn4v\") pod \"28d4f961-2402-4004-89af-6f24838dfcc8\" (UID: \"28d4f961-2402-4004-89af-6f24838dfcc8\") " Jul 06 22:31:17.196639 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.196615 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28d4f961-2402-4004-89af-6f24838dfcc8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "28d4f961-2402-4004-89af-6f24838dfcc8" (UID: "28d4f961-2402-4004-89af-6f24838dfcc8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:31:17.198277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.198255 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d4f961-2402-4004-89af-6f24838dfcc8-kube-api-access-lgn4v" (OuterVolumeSpecName: "kube-api-access-lgn4v") pod "28d4f961-2402-4004-89af-6f24838dfcc8" (UID: "28d4f961-2402-4004-89af-6f24838dfcc8"). InnerVolumeSpecName "kube-api-access-lgn4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:31:17.297309 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.297279 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28d4f961-2402-4004-89af-6f24838dfcc8-extensions-socket-volume\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:31:17.297309 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.297306 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgn4v\" (UniqueName: \"kubernetes.io/projected/28d4f961-2402-4004-89af-6f24838dfcc8-kube-api-access-lgn4v\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:31:17.873061 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.873028 2564 generic.go:356] "Generic (PLEG): container finished" podID="28d4f961-2402-4004-89af-6f24838dfcc8" containerID="c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d" exitCode=0 Jul 06 22:31:17.873533 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.873072 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" Jul 06 22:31:17.873533 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.873111 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" event={"ID":"28d4f961-2402-4004-89af-6f24838dfcc8","Type":"ContainerDied","Data":"c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d"} Jul 06 22:31:17.873533 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.873162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65" event={"ID":"28d4f961-2402-4004-89af-6f24838dfcc8","Type":"ContainerDied","Data":"f4d42ed5a140c69b41e8dd0ff5766dc23c82e0e36e2c22f21c2a384957ad2619"} Jul 06 22:31:17.873533 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.873185 2564 scope.go:117] "RemoveContainer" containerID="c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d" Jul 06 22:31:17.881949 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.881932 2564 scope.go:117] "RemoveContainer" containerID="c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d" Jul 06 22:31:17.882215 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:17.882196 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d\": container with ID starting with c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d not found: ID does not exist" containerID="c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d" Jul 06 22:31:17.882262 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.882225 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d"} err="failed to get container status \"c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d\": rpc error: code = NotFound desc = could not find container \"c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d\": container with ID starting with c056ee7e52788d77b67d79dbd4cab213a2322297bc6ef234de90cb4f6765ad0d not found: ID does not exist" Jul 06 22:31:17.900062 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.900032 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65"] Jul 06 22:31:17.910653 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:17.910624 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hkk65"] Jul 06 22:31:18.809864 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:18.809832 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28d4f961-2402-4004-89af-6f24838dfcc8" path="/var/lib/kubelet/pods/28d4f961-2402-4004-89af-6f24838dfcc8/volumes" Jul 06 22:31:23.620469 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:23.620334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:23.722884 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:23.722835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:26.196766 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:26.196735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:26.296745 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:26.296705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:33.270417 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.270380 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:33.339056 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.339018 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz"] Jul 06 22:31:33.339436 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.339407 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28d4f961-2402-4004-89af-6f24838dfcc8" containerName="manager" Jul 06 22:31:33.339502 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.339440 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="28d4f961-2402-4004-89af-6f24838dfcc8" containerName="manager" Jul 06 22:31:33.339540 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.339520 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="28d4f961-2402-4004-89af-6f24838dfcc8" containerName="manager" Jul 06 22:31:33.341579 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.341563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.344409 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.344387 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-pwfrs\"" Jul 06 22:31:33.359824 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.359796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz"] Jul 06 22:31:33.432831 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.432795 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:33.441921 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.441878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.441924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442238 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442238 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442149 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442238 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442238 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcq58\" (UniqueName: \"kubernetes.io/projected/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-kube-api-access-zcq58\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.442388 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.442254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.542897 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.542793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.542897 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.542844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.542962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcq58\" (UniqueName: \"kubernetes.io/projected/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-kube-api-access-zcq58\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543197 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543549 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543549 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543305 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543549 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543416 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543549 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543539 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543761 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543597 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.543971 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.543950 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.545492 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.545471 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.545739 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.545720 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.557102 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.557065 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcq58\" (UniqueName: \"kubernetes.io/projected/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-kube-api-access-zcq58\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.557802 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.557782 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3a0dda1-4e7e-45b4-9b66-4eef140d371b-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g8qkz\" (UID: \"e3a0dda1-4e7e-45b4-9b66-4eef140d371b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.571239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.571202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:33.653769 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.653731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:33.670781 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.670735 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:33.809103 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.809074 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz"] Jul 06 22:31:33.812015 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:31:33.811984 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a0dda1_4e7e_45b4_9b66_4eef140d371b.slice/crio-02c6d7af935697c5226ca1f68c479004183bd5bdc43a0b302c889eaa6ad08e88 WatchSource:0}: Error finding container 02c6d7af935697c5226ca1f68c479004183bd5bdc43a0b302c889eaa6ad08e88: Status 404 returned error can't find the container with id 02c6d7af935697c5226ca1f68c479004183bd5bdc43a0b302c889eaa6ad08e88 Jul 06 22:31:33.814069 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.814036 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:31:33.814163 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.814118 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:31:33.814231 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.814161 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:31:33.947693 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.947658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" event={"ID":"e3a0dda1-4e7e-45b4-9b66-4eef140d371b","Type":"ContainerStarted","Data":"78e65575d74f8dadac3decd1f23ae53b7c6d7374baebb046e46d261e4ebd759a"} Jul 06 22:31:33.947868 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.947723 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" event={"ID":"e3a0dda1-4e7e-45b4-9b66-4eef140d371b","Type":"ContainerStarted","Data":"02c6d7af935697c5226ca1f68c479004183bd5bdc43a0b302c889eaa6ad08e88"} Jul 06 22:31:33.981813 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:33.981758 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" podStartSLOduration=0.981738213 podStartE2EDuration="981.738213ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:33.98033803 +0000 UTC m=+455.716740021" watchObservedRunningTime="2026-07-06 22:31:33.981738213 +0000 UTC m=+455.718140200" Jul 06 22:31:34.654813 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:34.654780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:34.660046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:34.660019 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:34.952307 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:34.952229 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:34.953390 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:34.953369 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:31:35.197333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:35.197301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:35.347237 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:35.347195 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:35.448131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:35.448093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:36.094948 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.094912 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cxz2x"] Jul 06 22:31:36.097623 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.097600 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:36.102186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.102166 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-tczmh\"" Jul 06 22:31:36.117828 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.117805 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxz2x"] Jul 06 22:31:36.165877 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.165841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc88\" (UniqueName: \"kubernetes.io/projected/0b6af17c-66ce-48c5-af38-e749ec24db88-kube-api-access-hhc88\") pod \"authorino-58cd955675-cxz2x\" (UID: \"0b6af17c-66ce-48c5-af38-e749ec24db88\") " pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:36.168546 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.168517 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:36.266280 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.266246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc88\" (UniqueName: \"kubernetes.io/projected/0b6af17c-66ce-48c5-af38-e749ec24db88-kube-api-access-hhc88\") pod \"authorino-58cd955675-cxz2x\" (UID: \"0b6af17c-66ce-48c5-af38-e749ec24db88\") " pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:36.270786 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.270760 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:31:36.278547 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.278524 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc88\" (UniqueName: \"kubernetes.io/projected/0b6af17c-66ce-48c5-af38-e749ec24db88-kube-api-access-hhc88\") pod \"authorino-58cd955675-cxz2x\" (UID: \"0b6af17c-66ce-48c5-af38-e749ec24db88\") " pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:36.329077 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.329041 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-f2g7x"] Jul 06 22:31:36.331569 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.331547 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:31:36.341417 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.341386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f2g7x"] Jul 06 22:31:36.406744 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.406661 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:36.468024 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.467993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfc2z\" (UniqueName: \"kubernetes.io/projected/95259465-eaa4-462a-bfa6-1e65e5d253e1-kube-api-access-dfc2z\") pod \"authorino-75d556bd77-f2g7x\" (UID: \"95259465-eaa4-462a-bfa6-1e65e5d253e1\") " pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:31:36.550854 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.550831 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxz2x"] Jul 06 22:31:36.553242 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:31:36.553211 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b6af17c_66ce_48c5_af38_e749ec24db88.slice/crio-24f9217a0343927e602560e8eb7c83246a7f8108a10b7b5422a695ed5ab6a8ba WatchSource:0}: Error finding container 24f9217a0343927e602560e8eb7c83246a7f8108a10b7b5422a695ed5ab6a8ba: Status 404 returned error can't find the container with id 24f9217a0343927e602560e8eb7c83246a7f8108a10b7b5422a695ed5ab6a8ba Jul 06 22:31:36.568798 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.568772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfc2z\" (UniqueName: \"kubernetes.io/projected/95259465-eaa4-462a-bfa6-1e65e5d253e1-kube-api-access-dfc2z\") pod \"authorino-75d556bd77-f2g7x\" (UID: \"95259465-eaa4-462a-bfa6-1e65e5d253e1\") " pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:31:36.578298 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.578272 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfc2z\" (UniqueName: \"kubernetes.io/projected/95259465-eaa4-462a-bfa6-1e65e5d253e1-kube-api-access-dfc2z\") pod \"authorino-75d556bd77-f2g7x\" (UID: \"95259465-eaa4-462a-bfa6-1e65e5d253e1\") " pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:31:36.642482 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.642444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:31:36.787759 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.787723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f2g7x"] Jul 06 22:31:36.790513 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:31:36.790486 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod95259465_eaa4_462a_bfa6_1e65e5d253e1.slice/crio-67aa7f7abac9b62f6de62efcff8ccde2de3d72ec0f1e028f0b40ce01eadd2213 WatchSource:0}: Error finding container 67aa7f7abac9b62f6de62efcff8ccde2de3d72ec0f1e028f0b40ce01eadd2213: Status 404 returned error can't find the container with id 67aa7f7abac9b62f6de62efcff8ccde2de3d72ec0f1e028f0b40ce01eadd2213 Jul 06 22:31:36.962545 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.962463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxz2x" event={"ID":"0b6af17c-66ce-48c5-af38-e749ec24db88","Type":"ContainerStarted","Data":"24f9217a0343927e602560e8eb7c83246a7f8108a10b7b5422a695ed5ab6a8ba"} Jul 06 22:31:36.963671 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:36.963641 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f2g7x" event={"ID":"95259465-eaa4-462a-bfa6-1e65e5d253e1","Type":"ContainerStarted","Data":"67aa7f7abac9b62f6de62efcff8ccde2de3d72ec0f1e028f0b40ce01eadd2213"} Jul 06 22:31:39.984947 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:39.984907 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f2g7x" event={"ID":"95259465-eaa4-462a-bfa6-1e65e5d253e1","Type":"ContainerStarted","Data":"f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301"} Jul 06 22:31:39.986679 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:39.986642 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxz2x" event={"ID":"0b6af17c-66ce-48c5-af38-e749ec24db88","Type":"ContainerStarted","Data":"df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039"} Jul 06 22:31:40.006058 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:40.006002 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-f2g7x" podStartSLOduration=0.884819809 podStartE2EDuration="4.005986989s" totalImagesPullingTime="3.12116718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:40.004644877 +0000 UTC m=+461.741046864" watchObservedRunningTime="2026-07-06 22:31:40.005986989 +0000 UTC m=+461.742388976" Jul 06 22:31:40.028671 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:40.028051 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cxz2x" podStartSLOduration=0.681800775 podStartE2EDuration="4.028027762s" totalImagesPullingTime="3.346226987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:31:40.027513169 +0000 UTC m=+461.763915182" watchObservedRunningTime="2026-07-06 22:31:40.028027762 +0000 UTC m=+461.764429749" Jul 06 22:31:40.034826 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:40.034702 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxz2x"] Jul 06 22:31:41.997507 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:41.997465 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cxz2x" podUID="0b6af17c-66ce-48c5-af38-e749ec24db88" containerName="authorino" containerID="cri-o://df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039" gracePeriod=30 Jul 06 22:31:42.238377 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:42.238350 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:42.327413 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:42.327317 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhc88\" (UniqueName: \"kubernetes.io/projected/0b6af17c-66ce-48c5-af38-e749ec24db88-kube-api-access-hhc88\") pod \"0b6af17c-66ce-48c5-af38-e749ec24db88\" (UID: \"0b6af17c-66ce-48c5-af38-e749ec24db88\") " Jul 06 22:31:42.329396 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:42.329362 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b6af17c-66ce-48c5-af38-e749ec24db88-kube-api-access-hhc88" (OuterVolumeSpecName: "kube-api-access-hhc88") pod "0b6af17c-66ce-48c5-af38-e749ec24db88" (UID: "0b6af17c-66ce-48c5-af38-e749ec24db88"). InnerVolumeSpecName "kube-api-access-hhc88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:31:42.428065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:42.428025 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhc88\" (UniqueName: \"kubernetes.io/projected/0b6af17c-66ce-48c5-af38-e749ec24db88-kube-api-access-hhc88\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:31:43.004605 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.004566 2564 generic.go:356] "Generic (PLEG): container finished" podID="0b6af17c-66ce-48c5-af38-e749ec24db88" containerID="df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039" exitCode=0 Jul 06 22:31:43.005000 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.004621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxz2x" event={"ID":"0b6af17c-66ce-48c5-af38-e749ec24db88","Type":"ContainerDied","Data":"df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039"} Jul 06 22:31:43.005000 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.004637 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cxz2x" Jul 06 22:31:43.005000 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.004664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cxz2x" event={"ID":"0b6af17c-66ce-48c5-af38-e749ec24db88","Type":"ContainerDied","Data":"24f9217a0343927e602560e8eb7c83246a7f8108a10b7b5422a695ed5ab6a8ba"} Jul 06 22:31:43.005000 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.004681 2564 scope.go:117] "RemoveContainer" containerID="df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039" Jul 06 22:31:43.013336 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.013321 2564 scope.go:117] "RemoveContainer" containerID="df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039" Jul 06 22:31:43.013635 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:31:43.013614 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039\": container with ID starting with df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039 not found: ID does not exist" containerID="df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039" Jul 06 22:31:43.013700 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.013643 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039"} err="failed to get container status \"df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039\": rpc error: code = NotFound desc = could not find container \"df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039\": container with ID starting with df04642c44c56e4995df70851084acbeb9a6d95bc2c31ace47958fa09265c039 not found: ID does not exist" Jul 06 22:31:43.023022 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.022995 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxz2x"] Jul 06 22:31:43.026329 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:43.026306 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cxz2x"] Jul 06 22:31:44.810054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:44.810019 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b6af17c-66ce-48c5-af38-e749ec24db88" path="/var/lib/kubelet/pods/0b6af17c-66ce-48c5-af38-e749ec24db88/volumes" Jul 06 22:31:56.701141 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:31:56.701097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:08.896877 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:08.896820 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:09.123907 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.123874 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wvs69"] Jul 06 22:32:09.124402 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.124379 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b6af17c-66ce-48c5-af38-e749ec24db88" containerName="authorino" Jul 06 22:32:09.124494 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.124405 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b6af17c-66ce-48c5-af38-e749ec24db88" containerName="authorino" Jul 06 22:32:09.124541 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.124533 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b6af17c-66ce-48c5-af38-e749ec24db88" containerName="authorino" Jul 06 22:32:09.130879 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.130805 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:09.141160 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.141128 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wvs69"] Jul 06 22:32:09.166367 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.166286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxrsp\" (UniqueName: \"kubernetes.io/projected/f8937e34-2ecf-4ac4-b3c7-7651b1c35740-kube-api-access-mxrsp\") pod \"authorino-5d4b67c446-wvs69\" (UID: \"f8937e34-2ecf-4ac4-b3c7-7651b1c35740\") " pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:09.267597 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.267559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxrsp\" (UniqueName: \"kubernetes.io/projected/f8937e34-2ecf-4ac4-b3c7-7651b1c35740-kube-api-access-mxrsp\") pod \"authorino-5d4b67c446-wvs69\" (UID: \"f8937e34-2ecf-4ac4-b3c7-7651b1c35740\") " pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:09.277005 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.276972 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxrsp\" (UniqueName: \"kubernetes.io/projected/f8937e34-2ecf-4ac4-b3c7-7651b1c35740-kube-api-access-mxrsp\") pod \"authorino-5d4b67c446-wvs69\" (UID: \"f8937e34-2ecf-4ac4-b3c7-7651b1c35740\") " pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:09.336917 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.336876 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wvs69"] Jul 06 22:32:09.337190 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.337177 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:09.356813 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.356781 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55b596d4bb-t96sh"] Jul 06 22:32:09.360286 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.360265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:09.369105 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.368932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55b596d4bb-t96sh"] Jul 06 22:32:09.471149 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.471121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmwrh\" (UniqueName: \"kubernetes.io/projected/8f70bb96-820e-491a-92f6-150b67a886cc-kube-api-access-cmwrh\") pod \"authorino-55b596d4bb-t96sh\" (UID: \"8f70bb96-820e-491a-92f6-150b67a886cc\") " pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:09.484263 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.484240 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wvs69"] Jul 06 22:32:09.486411 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:32:09.486382 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8937e34_2ecf_4ac4_b3c7_7651b1c35740.slice/crio-6f9ae2128c63028d0e83c937326fbdb2397933882892472df6897688a7b307ac WatchSource:0}: Error finding container 6f9ae2128c63028d0e83c937326fbdb2397933882892472df6897688a7b307ac: Status 404 returned error can't find the container with id 6f9ae2128c63028d0e83c937326fbdb2397933882892472df6897688a7b307ac Jul 06 22:32:09.572085 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.572041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmwrh\" (UniqueName: \"kubernetes.io/projected/8f70bb96-820e-491a-92f6-150b67a886cc-kube-api-access-cmwrh\") pod \"authorino-55b596d4bb-t96sh\" (UID: \"8f70bb96-820e-491a-92f6-150b67a886cc\") " pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:09.580387 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.580350 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmwrh\" (UniqueName: \"kubernetes.io/projected/8f70bb96-820e-491a-92f6-150b67a886cc-kube-api-access-cmwrh\") pod \"authorino-55b596d4bb-t96sh\" (UID: \"8f70bb96-820e-491a-92f6-150b67a886cc\") " pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:09.587633 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.587602 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:09.672522 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.672489 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:09.756464 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.756414 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55b596d4bb-t96sh"] Jul 06 22:32:09.786410 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.786378 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f96d75cbb-bbnjx"] Jul 06 22:32:09.792334 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.792307 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:09.794690 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.794665 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 06 22:32:09.796288 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.796266 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f96d75cbb-bbnjx"] Jul 06 22:32:09.813577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.813528 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55b596d4bb-t96sh"] Jul 06 22:32:09.816439 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:32:09.816395 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f70bb96_820e_491a_92f6_150b67a886cc.slice/crio-061723b49f8c2ec2c5a4edd879ebc6172b09a46c1b59dec0ad2e853630a18316 WatchSource:0}: Error finding container 061723b49f8c2ec2c5a4edd879ebc6172b09a46c1b59dec0ad2e853630a18316: Status 404 returned error can't find the container with id 061723b49f8c2ec2c5a4edd879ebc6172b09a46c1b59dec0ad2e853630a18316 Jul 06 22:32:09.873941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.873906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7411982-5615-430e-83cb-79299302ebde-tls-cert\") pod \"authorino-5f96d75cbb-bbnjx\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:09.874137 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.873973 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzxgx\" (UniqueName: \"kubernetes.io/projected/c7411982-5615-430e-83cb-79299302ebde-kube-api-access-bzxgx\") pod \"authorino-5f96d75cbb-bbnjx\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:09.975097 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.975005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7411982-5615-430e-83cb-79299302ebde-tls-cert\") pod \"authorino-5f96d75cbb-bbnjx\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:09.975097 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.975061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzxgx\" (UniqueName: \"kubernetes.io/projected/c7411982-5615-430e-83cb-79299302ebde-kube-api-access-bzxgx\") pod \"authorino-5f96d75cbb-bbnjx\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:09.977705 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.977685 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7411982-5615-430e-83cb-79299302ebde-tls-cert\") pod \"authorino-5f96d75cbb-bbnjx\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:09.983975 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:09.983952 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzxgx\" (UniqueName: \"kubernetes.io/projected/c7411982-5615-430e-83cb-79299302ebde-kube-api-access-bzxgx\") pod \"authorino-5f96d75cbb-bbnjx\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:10.105207 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:10.105170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:32:10.147699 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:10.147661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b596d4bb-t96sh" event={"ID":"8f70bb96-820e-491a-92f6-150b67a886cc","Type":"ContainerStarted","Data":"061723b49f8c2ec2c5a4edd879ebc6172b09a46c1b59dec0ad2e853630a18316"} Jul 06 22:32:10.149031 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:10.149008 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wvs69" event={"ID":"f8937e34-2ecf-4ac4-b3c7-7651b1c35740","Type":"ContainerStarted","Data":"6f9ae2128c63028d0e83c937326fbdb2397933882892472df6897688a7b307ac"} Jul 06 22:32:10.276709 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:10.276404 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f96d75cbb-bbnjx"] Jul 06 22:32:10.279892 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:32:10.279545 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7411982_5615_430e_83cb_79299302ebde.slice/crio-89de8012c32408823381536ddb4e62bbedd6f68e46f062e22124af77ab0501d2 WatchSource:0}: Error finding container 89de8012c32408823381536ddb4e62bbedd6f68e46f062e22124af77ab0501d2: Status 404 returned error can't find the container with id 89de8012c32408823381536ddb4e62bbedd6f68e46f062e22124af77ab0501d2 Jul 06 22:32:11.155981 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.155947 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b596d4bb-t96sh" event={"ID":"8f70bb96-820e-491a-92f6-150b67a886cc","Type":"ContainerStarted","Data":"9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d"} Jul 06 22:32:11.156486 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.156010 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-55b596d4bb-t96sh" podUID="8f70bb96-820e-491a-92f6-150b67a886cc" containerName="authorino" containerID="cri-o://9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d" gracePeriod=30 Jul 06 22:32:11.157695 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.157665 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-wvs69" podUID="f8937e34-2ecf-4ac4-b3c7-7651b1c35740" containerName="authorino" containerID="cri-o://4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d" gracePeriod=30 Jul 06 22:32:11.157809 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.157669 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wvs69" event={"ID":"f8937e34-2ecf-4ac4-b3c7-7651b1c35740","Type":"ContainerStarted","Data":"4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d"} Jul 06 22:32:11.159394 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.159364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" event={"ID":"c7411982-5615-430e-83cb-79299302ebde","Type":"ContainerStarted","Data":"1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f"} Jul 06 22:32:11.159508 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.159407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" event={"ID":"c7411982-5615-430e-83cb-79299302ebde","Type":"ContainerStarted","Data":"89de8012c32408823381536ddb4e62bbedd6f68e46f062e22124af77ab0501d2"} Jul 06 22:32:11.173603 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.173543 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55b596d4bb-t96sh" podStartSLOduration=1.833775216 podStartE2EDuration="2.173517379s" totalImagesPullingTime="339.742163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:11.173056344 +0000 UTC m=+492.909458332" watchObservedRunningTime="2026-07-06 22:32:11.173517379 +0000 UTC m=+492.909919370" Jul 06 22:32:11.190915 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.190849 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-wvs69" podStartSLOduration=1.566447225 podStartE2EDuration="2.190829019s" totalImagesPullingTime="624.381794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:11.190766631 +0000 UTC m=+492.927168615" watchObservedRunningTime="2026-07-06 22:32:11.190829019 +0000 UTC m=+492.927231006" Jul 06 22:32:11.208053 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.208000 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" podStartSLOduration=1.8864672420000002 podStartE2EDuration="2.20798246s" totalImagesPullingTime="321.515218ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:32:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:32:11.206537782 +0000 UTC m=+492.942939769" watchObservedRunningTime="2026-07-06 22:32:11.20798246 +0000 UTC m=+492.944384447" Jul 06 22:32:11.236529 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.236495 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f2g7x"] Jul 06 22:32:11.236736 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.236683 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-f2g7x" podUID="95259465-eaa4-462a-bfa6-1e65e5d253e1" containerName="authorino" containerID="cri-o://f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301" gracePeriod=30 Jul 06 22:32:11.544623 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.544595 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:11.574717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.574685 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:11.590855 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.590810 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:32:11.593388 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.593360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmwrh\" (UniqueName: \"kubernetes.io/projected/8f70bb96-820e-491a-92f6-150b67a886cc-kube-api-access-cmwrh\") pod \"8f70bb96-820e-491a-92f6-150b67a886cc\" (UID: \"8f70bb96-820e-491a-92f6-150b67a886cc\") " Jul 06 22:32:11.595458 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.595406 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f70bb96-820e-491a-92f6-150b67a886cc-kube-api-access-cmwrh" (OuterVolumeSpecName: "kube-api-access-cmwrh") pod "8f70bb96-820e-491a-92f6-150b67a886cc" (UID: "8f70bb96-820e-491a-92f6-150b67a886cc"). InnerVolumeSpecName "kube-api-access-cmwrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:32:11.694150 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.694059 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxrsp\" (UniqueName: \"kubernetes.io/projected/f8937e34-2ecf-4ac4-b3c7-7651b1c35740-kube-api-access-mxrsp\") pod \"f8937e34-2ecf-4ac4-b3c7-7651b1c35740\" (UID: \"f8937e34-2ecf-4ac4-b3c7-7651b1c35740\") " Jul 06 22:32:11.694150 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.694114 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfc2z\" (UniqueName: \"kubernetes.io/projected/95259465-eaa4-462a-bfa6-1e65e5d253e1-kube-api-access-dfc2z\") pod \"95259465-eaa4-462a-bfa6-1e65e5d253e1\" (UID: \"95259465-eaa4-462a-bfa6-1e65e5d253e1\") " Jul 06 22:32:11.694376 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.694284 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmwrh\" (UniqueName: \"kubernetes.io/projected/8f70bb96-820e-491a-92f6-150b67a886cc-kube-api-access-cmwrh\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:32:11.696188 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.696156 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8937e34-2ecf-4ac4-b3c7-7651b1c35740-kube-api-access-mxrsp" (OuterVolumeSpecName: "kube-api-access-mxrsp") pod "f8937e34-2ecf-4ac4-b3c7-7651b1c35740" (UID: "f8937e34-2ecf-4ac4-b3c7-7651b1c35740"). InnerVolumeSpecName "kube-api-access-mxrsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:32:11.696188 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.696173 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95259465-eaa4-462a-bfa6-1e65e5d253e1-kube-api-access-dfc2z" (OuterVolumeSpecName: "kube-api-access-dfc2z") pod "95259465-eaa4-462a-bfa6-1e65e5d253e1" (UID: "95259465-eaa4-462a-bfa6-1e65e5d253e1"). InnerVolumeSpecName "kube-api-access-dfc2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:32:11.795784 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.795738 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxrsp\" (UniqueName: \"kubernetes.io/projected/f8937e34-2ecf-4ac4-b3c7-7651b1c35740-kube-api-access-mxrsp\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:32:11.795784 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:11.795775 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfc2z\" (UniqueName: \"kubernetes.io/projected/95259465-eaa4-462a-bfa6-1e65e5d253e1-kube-api-access-dfc2z\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:32:12.165693 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.165656 2564 generic.go:356] "Generic (PLEG): container finished" podID="8f70bb96-820e-491a-92f6-150b67a886cc" containerID="9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d" exitCode=0 Jul 06 22:32:12.166170 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.165700 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55b596d4bb-t96sh" Jul 06 22:32:12.166170 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.165740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b596d4bb-t96sh" event={"ID":"8f70bb96-820e-491a-92f6-150b67a886cc","Type":"ContainerDied","Data":"9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d"} Jul 06 22:32:12.166170 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.165793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55b596d4bb-t96sh" event={"ID":"8f70bb96-820e-491a-92f6-150b67a886cc","Type":"ContainerDied","Data":"061723b49f8c2ec2c5a4edd879ebc6172b09a46c1b59dec0ad2e853630a18316"} Jul 06 22:32:12.166170 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.165811 2564 scope.go:117] "RemoveContainer" containerID="9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d" Jul 06 22:32:12.167194 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.167132 2564 generic.go:356] "Generic (PLEG): container finished" podID="f8937e34-2ecf-4ac4-b3c7-7651b1c35740" containerID="4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d" exitCode=0 Jul 06 22:32:12.167194 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.167177 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-wvs69" Jul 06 22:32:12.167194 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.167184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wvs69" event={"ID":"f8937e34-2ecf-4ac4-b3c7-7651b1c35740","Type":"ContainerDied","Data":"4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d"} Jul 06 22:32:12.167385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.167221 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-wvs69" event={"ID":"f8937e34-2ecf-4ac4-b3c7-7651b1c35740","Type":"ContainerDied","Data":"6f9ae2128c63028d0e83c937326fbdb2397933882892472df6897688a7b307ac"} Jul 06 22:32:12.168649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.168629 2564 generic.go:356] "Generic (PLEG): container finished" podID="95259465-eaa4-462a-bfa6-1e65e5d253e1" containerID="f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301" exitCode=0 Jul 06 22:32:12.168758 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.168701 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-f2g7x" Jul 06 22:32:12.168758 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.168738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f2g7x" event={"ID":"95259465-eaa4-462a-bfa6-1e65e5d253e1","Type":"ContainerDied","Data":"f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301"} Jul 06 22:32:12.168879 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.168771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-f2g7x" event={"ID":"95259465-eaa4-462a-bfa6-1e65e5d253e1","Type":"ContainerDied","Data":"67aa7f7abac9b62f6de62efcff8ccde2de3d72ec0f1e028f0b40ce01eadd2213"} Jul 06 22:32:12.176852 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.176835 2564 scope.go:117] "RemoveContainer" containerID="9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d" Jul 06 22:32:12.177129 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:32:12.177110 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d\": container with ID starting with 9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d not found: ID does not exist" containerID="9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d" Jul 06 22:32:12.177204 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.177136 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d"} err="failed to get container status \"9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d\": rpc error: code = NotFound desc = could not find container \"9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d\": container with ID starting with 9e77baf5fa35cbb3360c55e652a7a9b83db62d2076b811d298bf6aee6594230d not found: ID does not exist" Jul 06 22:32:12.177204 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.177152 2564 scope.go:117] "RemoveContainer" containerID="4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d" Jul 06 22:32:12.185520 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.185502 2564 scope.go:117] "RemoveContainer" containerID="4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d" Jul 06 22:32:12.185756 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:32:12.185736 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d\": container with ID starting with 4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d not found: ID does not exist" containerID="4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d" Jul 06 22:32:12.185817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.185763 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d"} err="failed to get container status \"4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d\": rpc error: code = NotFound desc = could not find container \"4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d\": container with ID starting with 4d94d21bc1114437e6365d68f1b843d332a2a115c719e215a5237da8534cb60d not found: ID does not exist" Jul 06 22:32:12.185817 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.185779 2564 scope.go:117] "RemoveContainer" containerID="f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301" Jul 06 22:32:12.189083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.189059 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55b596d4bb-t96sh"] Jul 06 22:32:12.193770 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.193752 2564 scope.go:117] "RemoveContainer" containerID="f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301" Jul 06 22:32:12.194045 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:32:12.194028 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301\": container with ID starting with f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301 not found: ID does not exist" containerID="f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301" Jul 06 22:32:12.194094 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.194052 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301"} err="failed to get container status \"f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301\": rpc error: code = NotFound desc = could not find container \"f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301\": container with ID starting with f43a6f3ffc5504f81de5d763eed53d88252a509b4cb60ffb601767bb74bee301 not found: ID does not exist" Jul 06 22:32:12.194938 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.194921 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55b596d4bb-t96sh"] Jul 06 22:32:12.204868 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.204839 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wvs69"] Jul 06 22:32:12.208607 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.208581 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-wvs69"] Jul 06 22:32:12.218218 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.218190 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f2g7x"] Jul 06 22:32:12.221856 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.221833 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-f2g7x"] Jul 06 22:32:12.809991 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.809962 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f70bb96-820e-491a-92f6-150b67a886cc" path="/var/lib/kubelet/pods/8f70bb96-820e-491a-92f6-150b67a886cc/volumes" Jul 06 22:32:12.810355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.810341 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95259465-eaa4-462a-bfa6-1e65e5d253e1" path="/var/lib/kubelet/pods/95259465-eaa4-462a-bfa6-1e65e5d253e1/volumes" Jul 06 22:32:12.810656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:12.810644 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8937e34-2ecf-4ac4-b3c7-7651b1c35740" path="/var/lib/kubelet/pods/f8937e34-2ecf-4ac4-b3c7-7651b1c35740/volumes" Jul 06 22:32:15.504350 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:15.504317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:15.669297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:15.669255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:15.734669 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:15.734623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:18.955868 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:18.955821 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:19.120120 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:19.120084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:51.396793 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:51.396663 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:51.924757 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:51.924711 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:52.030850 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:52.030819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:52.317156 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:52.317081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:52.420887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:52.420848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:32:52.516523 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:32:52.516479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:00.413501 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:00.413467 2564 generic.go:356] "Generic (PLEG): container finished" podID="e3a0dda1-4e7e-45b4-9b66-4eef140d371b" containerID="78e65575d74f8dadac3decd1f23ae53b7c6d7374baebb046e46d261e4ebd759a" exitCode=0 Jul 06 22:33:00.413501 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:00.413506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" event={"ID":"e3a0dda1-4e7e-45b4-9b66-4eef140d371b","Type":"ContainerDied","Data":"78e65575d74f8dadac3decd1f23ae53b7c6d7374baebb046e46d261e4ebd759a"} Jul 06 22:33:00.414026 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:00.413839 2564 scope.go:117] "RemoveContainer" containerID="78e65575d74f8dadac3decd1f23ae53b7c6d7374baebb046e46d261e4ebd759a" Jul 06 22:33:00.414786 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:00.414759 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:33:00.414848 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:00.414825 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:33:00.414888 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:00.414854 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:33:01.420136 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:01.420108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" event={"ID":"e3a0dda1-4e7e-45b4-9b66-4eef140d371b","Type":"ContainerStarted","Data":"8b5db9f24cedadbfe21d5b5ffdb49102cdcce9417eb582252c42011fe10bf175"} Jul 06 22:33:01.654356 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:01.654319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:33:01.655531 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:01.655495 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g8qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:33:01.655653 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:01.655548 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" podUID="e3a0dda1-4e7e-45b4-9b66-4eef140d371b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 22:33:02.654735 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:02.654701 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g8qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:33:02.655139 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:02.654756 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" podUID="e3a0dda1-4e7e-45b4-9b66-4eef140d371b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 22:33:03.186114 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:03.186050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:03.276848 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:03.276800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:03.654733 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:03.654696 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g8qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:33:03.654919 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:03.654751 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" podUID="e3a0dda1-4e7e-45b4-9b66-4eef140d371b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 22:33:04.654752 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:04.654714 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g8qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:33:04.654951 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:04.654786 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" podUID="e3a0dda1-4e7e-45b4-9b66-4eef140d371b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 22:33:04.953050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:04.952950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:33:05.654543 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:05.654511 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g8qkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 22:33:05.654740 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:05.654565 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" podUID="e3a0dda1-4e7e-45b4-9b66-4eef140d371b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 22:33:06.657734 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:06.657694 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:33:06.658609 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:06.658590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g8qkz" Jul 06 22:33:32.117099 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117065 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2"] Jul 06 22:33:32.117718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117616 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95259465-eaa4-462a-bfa6-1e65e5d253e1" containerName="authorino" Jul 06 22:33:32.117718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117637 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="95259465-eaa4-462a-bfa6-1e65e5d253e1" containerName="authorino" Jul 06 22:33:32.117718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117653 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f70bb96-820e-491a-92f6-150b67a886cc" containerName="authorino" Jul 06 22:33:32.117718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117663 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f70bb96-820e-491a-92f6-150b67a886cc" containerName="authorino" Jul 06 22:33:32.117718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117687 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8937e34-2ecf-4ac4-b3c7-7651b1c35740" containerName="authorino" Jul 06 22:33:32.117718 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117696 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8937e34-2ecf-4ac4-b3c7-7651b1c35740" containerName="authorino" Jul 06 22:33:32.118014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117771 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="95259465-eaa4-462a-bfa6-1e65e5d253e1" containerName="authorino" Jul 06 22:33:32.118014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117788 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8937e34-2ecf-4ac4-b3c7-7651b1c35740" containerName="authorino" Jul 06 22:33:32.118014 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.117798 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f70bb96-820e-491a-92f6-150b67a886cc" containerName="authorino" Jul 06 22:33:32.119852 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.119830 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.122413 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.122388 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 22:33:32.123623 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.123601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w2bxs\"" Jul 06 22:33:32.123842 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.123828 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 22:33:32.123899 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.123840 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 22:33:32.131478 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.131452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2"] Jul 06 22:33:32.150621 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.150590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:32.217920 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.217883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.217920 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.217924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.218169 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.217949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96l47\" (UniqueName: \"kubernetes.io/projected/4a462b2d-2944-41c7-9531-38cb4dc59d6d-kube-api-access-96l47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.218169 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.218013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.218169 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.218064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.218169 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.218095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a462b2d-2944-41c7-9531-38cb4dc59d6d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.319546 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.319500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96l47\" (UniqueName: \"kubernetes.io/projected/4a462b2d-2944-41c7-9531-38cb4dc59d6d-kube-api-access-96l47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.319771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.319556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.319771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.319595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.319771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.319631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a462b2d-2944-41c7-9531-38cb4dc59d6d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.319771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.319680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.319771 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.319699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.320054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.320032 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.320101 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.320075 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.320101 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.320093 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.322026 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.322005 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a462b2d-2944-41c7-9531-38cb4dc59d6d-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.322289 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.322273 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a462b2d-2944-41c7-9531-38cb4dc59d6d-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.331360 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.331337 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96l47\" (UniqueName: \"kubernetes.io/projected/4a462b2d-2944-41c7-9531-38cb4dc59d6d-kube-api-access-96l47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2\" (UID: \"4a462b2d-2944-41c7-9531-38cb4dc59d6d\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.431114 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.431016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:32.570996 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:32.570972 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2"] Jul 06 22:33:32.573205 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:33:32.573177 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a462b2d_2944_41c7_9531_38cb4dc59d6d.slice/crio-28342c934551fcd1e07157d0b17e1c88cc42ada9662b3a6e174717a2813daa76 WatchSource:0}: Error finding container 28342c934551fcd1e07157d0b17e1c88cc42ada9662b3a6e174717a2813daa76: Status 404 returned error can't find the container with id 28342c934551fcd1e07157d0b17e1c88cc42ada9662b3a6e174717a2813daa76 Jul 06 22:33:33.566792 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:33.566745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" event={"ID":"4a462b2d-2944-41c7-9531-38cb4dc59d6d","Type":"ContainerStarted","Data":"28342c934551fcd1e07157d0b17e1c88cc42ada9662b3a6e174717a2813daa76"} Jul 06 22:33:34.573260 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:34.573224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" event={"ID":"4a462b2d-2944-41c7-9531-38cb4dc59d6d","Type":"ContainerStarted","Data":"b2c9dc4c8246f76902b900e1673834ffac93a27444aaa7e601b3b1df29645707"} Jul 06 22:33:34.593700 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:34.593650 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" podStartSLOduration=1.42759665 podStartE2EDuration="2.593633197s" totalImagesPullingTime="1.166036547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:33:34.592128543 +0000 UTC m=+576.328530554" watchObservedRunningTime="2026-07-06 22:33:34.593633197 +0000 UTC m=+576.330035184" Jul 06 22:33:42.431988 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:42.431940 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:42.432446 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:42.432015 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:42.444439 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:42.444406 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:42.620701 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:42.620669 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2" Jul 06 22:33:43.128118 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:43.128077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:58.045849 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:58.045814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:58.266799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:58.266766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:58.374691 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:58.374665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:33:58.681028 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:58.680946 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:33:58.696558 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:33:58.696532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:34:12.179515 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:12.179479 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:24.333384 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.333303 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr"] Jul 06 22:34:24.336557 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.336539 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.339492 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.339469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 22:34:24.355583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.355555 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr"] Jul 06 22:34:24.380624 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.380596 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.380774 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.380646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.380774 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.380745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.380900 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.380825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.380900 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.380855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpmxr\" (UniqueName: \"kubernetes.io/projected/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-kube-api-access-hpmxr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.380900 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.380891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.411294 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.411246 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:24.482348 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.482562 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpmxr\" (UniqueName: \"kubernetes.io/projected/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-kube-api-access-hpmxr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.482562 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.482562 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.482717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.482717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482645 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.482959 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482938 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.483025 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.482997 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.483025 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.483009 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.484727 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.484697 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.484996 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.484974 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.491543 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.491518 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpmxr\" (UniqueName: \"kubernetes.io/projected/7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2-kube-api-access-hpmxr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-crfsr\" (UID: \"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.646755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.646657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:24.793334 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.793308 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr"] Jul 06 22:34:24.796368 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:34:24.796326 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c875ba1_564a_4e2a_bd4a_e7bb1fa93ff2.slice/crio-86fa153feac517a5449e44d64ad9cb350a36d26733c8e09c882aafb568328190 WatchSource:0}: Error finding container 86fa153feac517a5449e44d64ad9cb350a36d26733c8e09c882aafb568328190: Status 404 returned error can't find the container with id 86fa153feac517a5449e44d64ad9cb350a36d26733c8e09c882aafb568328190 Jul 06 22:34:24.797711 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.797689 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 22:34:24.809857 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:24.809820 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" event={"ID":"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2","Type":"ContainerStarted","Data":"86fa153feac517a5449e44d64ad9cb350a36d26733c8e09c882aafb568328190"} Jul 06 22:34:25.814542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:25.814504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" event={"ID":"7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2","Type":"ContainerStarted","Data":"0ddabfb8bb33925e7f2a436b973310b2a51b0ebdfd7d74d314da3ec3f941283e"} Jul 06 22:34:25.834708 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:25.834652 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" podStartSLOduration=1.65392925 podStartE2EDuration="1.834634422s" totalImagesPullingTime="180.705172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:25.83318799 +0000 UTC m=+627.569589979" watchObservedRunningTime="2026-07-06 22:34:25.834634422 +0000 UTC m=+627.571036409" Jul 06 22:34:34.646990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:34.646949 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:34.647485 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:34.647037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:34.659679 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:34.659655 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:34.867036 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:34.867004 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-crfsr" Jul 06 22:34:36.314340 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.314303 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6"] Jul 06 22:34:36.318216 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.318190 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.320839 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.320817 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 06 22:34:36.329000 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.328974 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6"] Jul 06 22:34:36.388249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.388218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.388249 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.388256 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6g5m\" (UniqueName: \"kubernetes.io/projected/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-kube-api-access-w6g5m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.388537 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.388289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.388537 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.388322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.388537 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.388366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.388537 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.388499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.469653 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.469616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:34:36.489007 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.488978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489009 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6g5m\" (UniqueName: \"kubernetes.io/projected/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-kube-api-access-w6g5m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489076 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489517 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489496 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489735 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489514 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.489735 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.489581 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.491348 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.491321 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.491648 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.491630 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.497960 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.497937 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6g5m\" (UniqueName: \"kubernetes.io/projected/ffbe7298-4203-4be7-9184-9eb5bb4e27d8-kube-api-access-w6g5m\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6\" (UID: \"ffbe7298-4203-4be7-9184-9eb5bb4e27d8\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.628361 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.628320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:36.785122 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.785088 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6"] Jul 06 22:34:36.788527 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:34:36.788499 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffbe7298_4203_4be7_9184_9eb5bb4e27d8.slice/crio-7a657edb29933558da2915ff6d0b88a08e87c1e899cdaaa41e92da09a08d1fa0 WatchSource:0}: Error finding container 7a657edb29933558da2915ff6d0b88a08e87c1e899cdaaa41e92da09a08d1fa0: Status 404 returned error can't find the container with id 7a657edb29933558da2915ff6d0b88a08e87c1e899cdaaa41e92da09a08d1fa0 Jul 06 22:34:36.867592 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:36.867556 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" event={"ID":"ffbe7298-4203-4be7-9184-9eb5bb4e27d8","Type":"ContainerStarted","Data":"7a657edb29933558da2915ff6d0b88a08e87c1e899cdaaa41e92da09a08d1fa0"} Jul 06 22:34:37.874214 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:37.874178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" event={"ID":"ffbe7298-4203-4be7-9184-9eb5bb4e27d8","Type":"ContainerStarted","Data":"de44daf42ec446adbd7fbc98d04264c37bad68dbc0acebde4fa7ff14d2827cb0"} Jul 06 22:34:37.894467 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:37.894391 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" podStartSLOduration=1.703183696 podStartE2EDuration="1.89437186s" totalImagesPullingTime="191.188164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:34:37.892473811 +0000 UTC m=+639.628875795" watchObservedRunningTime="2026-07-06 22:34:37.89437186 +0000 UTC m=+639.630773848" Jul 06 22:34:46.629246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:46.629191 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:46.629743 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:46.629266 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:46.642125 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:46.642093 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:34:46.933448 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:34:46.933348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6" Jul 06 22:35:15.375627 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:15.375583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:35:30.222568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.222528 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6989dd7f-46trd"] Jul 06 22:35:30.225870 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.225853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.234075 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.234048 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6989dd7f-46trd"] Jul 06 22:35:30.286999 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.286956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a89c4d84-97d0-4b6d-b755-09c131839a2e-tls-cert\") pod \"authorino-6f6989dd7f-46trd\" (UID: \"a89c4d84-97d0-4b6d-b755-09c131839a2e\") " pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.286999 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.286995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcrd\" (UniqueName: \"kubernetes.io/projected/a89c4d84-97d0-4b6d-b755-09c131839a2e-kube-api-access-4pcrd\") pod \"authorino-6f6989dd7f-46trd\" (UID: \"a89c4d84-97d0-4b6d-b755-09c131839a2e\") " pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.388495 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.388454 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a89c4d84-97d0-4b6d-b755-09c131839a2e-tls-cert\") pod \"authorino-6f6989dd7f-46trd\" (UID: \"a89c4d84-97d0-4b6d-b755-09c131839a2e\") " pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.388495 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.388496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcrd\" (UniqueName: \"kubernetes.io/projected/a89c4d84-97d0-4b6d-b755-09c131839a2e-kube-api-access-4pcrd\") pod \"authorino-6f6989dd7f-46trd\" (UID: \"a89c4d84-97d0-4b6d-b755-09c131839a2e\") " pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.390952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.390923 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a89c4d84-97d0-4b6d-b755-09c131839a2e-tls-cert\") pod \"authorino-6f6989dd7f-46trd\" (UID: \"a89c4d84-97d0-4b6d-b755-09c131839a2e\") " pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.396528 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.396505 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcrd\" (UniqueName: \"kubernetes.io/projected/a89c4d84-97d0-4b6d-b755-09c131839a2e-kube-api-access-4pcrd\") pod \"authorino-6f6989dd7f-46trd\" (UID: \"a89c4d84-97d0-4b6d-b755-09c131839a2e\") " pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.535594 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.535499 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6989dd7f-46trd" Jul 06 22:35:30.678690 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:30.678662 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6989dd7f-46trd"] Jul 06 22:35:30.681496 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:35:30.681469 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda89c4d84_97d0_4b6d_b755_09c131839a2e.slice/crio-a3eace5b7e5a95666698c1b2bb91f9a156606b9875ca05e45c14590facb98645 WatchSource:0}: Error finding container a3eace5b7e5a95666698c1b2bb91f9a156606b9875ca05e45c14590facb98645: Status 404 returned error can't find the container with id a3eace5b7e5a95666698c1b2bb91f9a156606b9875ca05e45c14590facb98645 Jul 06 22:35:31.148588 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.148538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6989dd7f-46trd" event={"ID":"a89c4d84-97d0-4b6d-b755-09c131839a2e","Type":"ContainerStarted","Data":"27d4e10ca2a9468b6a1d8a93f68920df292dff8cf25794266b7a1310eaf73103"} Jul 06 22:35:31.148811 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.148612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6989dd7f-46trd" event={"ID":"a89c4d84-97d0-4b6d-b755-09c131839a2e","Type":"ContainerStarted","Data":"a3eace5b7e5a95666698c1b2bb91f9a156606b9875ca05e45c14590facb98645"} Jul 06 22:35:31.167470 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.167389 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6989dd7f-46trd" podStartSLOduration=0.828994922 podStartE2EDuration="1.167365661s" totalImagesPullingTime="338.370739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:35:31.165539594 +0000 UTC m=+692.901941582" watchObservedRunningTime="2026-07-06 22:35:31.167365661 +0000 UTC m=+692.903767646" Jul 06 22:35:31.193657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.193598 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f96d75cbb-bbnjx"] Jul 06 22:35:31.193952 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.193920 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" podUID="c7411982-5615-430e-83cb-79299302ebde" containerName="authorino" containerID="cri-o://1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f" gracePeriod=30 Jul 06 22:35:31.473698 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.473647 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:35:31.602276 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.602240 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7411982-5615-430e-83cb-79299302ebde-tls-cert\") pod \"c7411982-5615-430e-83cb-79299302ebde\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " Jul 06 22:35:31.602491 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.602338 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzxgx\" (UniqueName: \"kubernetes.io/projected/c7411982-5615-430e-83cb-79299302ebde-kube-api-access-bzxgx\") pod \"c7411982-5615-430e-83cb-79299302ebde\" (UID: \"c7411982-5615-430e-83cb-79299302ebde\") " Jul 06 22:35:31.604370 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.604334 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7411982-5615-430e-83cb-79299302ebde-kube-api-access-bzxgx" (OuterVolumeSpecName: "kube-api-access-bzxgx") pod "c7411982-5615-430e-83cb-79299302ebde" (UID: "c7411982-5615-430e-83cb-79299302ebde"). InnerVolumeSpecName "kube-api-access-bzxgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:35:31.613802 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.613770 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7411982-5615-430e-83cb-79299302ebde-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c7411982-5615-430e-83cb-79299302ebde" (UID: "c7411982-5615-430e-83cb-79299302ebde"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:35:31.703180 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.703140 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c7411982-5615-430e-83cb-79299302ebde-tls-cert\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:35:31.703180 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.703176 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzxgx\" (UniqueName: \"kubernetes.io/projected/c7411982-5615-430e-83cb-79299302ebde-kube-api-access-bzxgx\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:35:31.722906 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.722811 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:35:31.815975 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:31.815941 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:35:32.156184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.156143 2564 generic.go:356] "Generic (PLEG): container finished" podID="c7411982-5615-430e-83cb-79299302ebde" containerID="1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f" exitCode=0 Jul 06 22:35:32.156385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.156190 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" Jul 06 22:35:32.156385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.156234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" event={"ID":"c7411982-5615-430e-83cb-79299302ebde","Type":"ContainerDied","Data":"1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f"} Jul 06 22:35:32.156385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.156285 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f96d75cbb-bbnjx" event={"ID":"c7411982-5615-430e-83cb-79299302ebde","Type":"ContainerDied","Data":"89de8012c32408823381536ddb4e62bbedd6f68e46f062e22124af77ab0501d2"} Jul 06 22:35:32.156385 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.156307 2564 scope.go:117] "RemoveContainer" containerID="1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f" Jul 06 22:35:32.166341 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.166317 2564 scope.go:117] "RemoveContainer" containerID="1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f" Jul 06 22:35:32.166698 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:35:32.166676 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f\": container with ID starting with 1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f not found: ID does not exist" containerID="1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f" Jul 06 22:35:32.166765 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.166709 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f"} err="failed to get container status \"1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f\": rpc error: code = NotFound desc = could not find container \"1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f\": container with ID starting with 1b07ea25071a60ddb6e93d6bf0938290313ca57d7fb30072fe4155673f5d033f not found: ID does not exist" Jul 06 22:35:32.179591 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.179555 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f96d75cbb-bbnjx"] Jul 06 22:35:32.185062 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.185039 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f96d75cbb-bbnjx"] Jul 06 22:35:32.812003 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.811968 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7411982-5615-430e-83cb-79299302ebde" path="/var/lib/kubelet/pods/c7411982-5615-430e-83cb-79299302ebde/volumes" Jul 06 22:35:32.829884 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.829850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:35:32.979158 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:35:32.979126 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:26.491372 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:26.491339 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:29.377800 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:29.377757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:29.478676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:29.478636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:37.545717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:37.545617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:42.708266 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:42.708235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:42.769493 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:42.769460 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:44.631810 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:44.631780 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:50.394033 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:50.393997 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:36:58.756267 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:36:58.756233 2564 scope.go:117] "RemoveContainer" containerID="f59cf17539f587653be902da1b0eb2db2ca7f3024800807d49a9a97e2f435bdc" Jul 06 22:37:28.183581 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:28.183537 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:28.281582 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:28.281538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:28.400372 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:28.400341 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:34.396412 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:34.396372 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:34.462699 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:34.462661 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:34.545162 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:34.545132 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:34.644746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:34.644714 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:34.745192 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:34.745109 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:40.644391 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:40.644336 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:40.748547 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:40.748516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:55.068465 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:55.068411 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:37:55.169871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:37:55.169837 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:38:58.709495 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:38:58.709390 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:38:58.722199 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:38:58.722174 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:39:30.531969 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:39:30.531932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:39:31.875086 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:39:31.875048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:41:07.457054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:07.457022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:41:07.512555 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:07.512526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:41:29.385270 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:29.385234 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68"] Jul 06 22:41:29.385780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:29.385477 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" podUID="4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" containerName="manager" containerID="cri-o://05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a" gracePeriod=10 Jul 06 22:41:32.353737 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.353705 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:41:32.468928 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.468830 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx5vw\" (UniqueName: \"kubernetes.io/projected/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-kube-api-access-rx5vw\") pod \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " Jul 06 22:41:32.468928 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.468902 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-extensions-socket-volume\") pod \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\" (UID: \"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15\") " Jul 06 22:41:32.469276 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.469253 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" (UID: "4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:41:32.471031 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.471005 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-kube-api-access-rx5vw" (OuterVolumeSpecName: "kube-api-access-rx5vw") pod "4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" (UID: "4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15"). InnerVolumeSpecName "kube-api-access-rx5vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:41:32.570536 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.570505 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-extensions-socket-volume\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:41:32.570536 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.570531 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5vw\" (UniqueName: \"kubernetes.io/projected/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15-kube-api-access-rx5vw\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:41:32.941867 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.941836 2564 generic.go:356] "Generic (PLEG): container finished" podID="4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" containerID="05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a" exitCode=0 Jul 06 22:41:32.942054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.941874 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" Jul 06 22:41:32.942054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.941921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" event={"ID":"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15","Type":"ContainerDied","Data":"05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a"} Jul 06 22:41:32.942054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.941975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68" event={"ID":"4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15","Type":"ContainerDied","Data":"63d413b4ce250de06464ee1b50f01e3114cafac4d176c5678d16b74c5daeaa96"} Jul 06 22:41:32.942054 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.941999 2564 scope.go:117] "RemoveContainer" containerID="05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a" Jul 06 22:41:32.951243 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.951228 2564 scope.go:117] "RemoveContainer" containerID="05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a" Jul 06 22:41:32.951513 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:41:32.951494 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a\": container with ID starting with 05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a not found: ID does not exist" containerID="05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a" Jul 06 22:41:32.951577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.951524 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a"} err="failed to get container status \"05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a\": rpc error: code = NotFound desc = could not find container \"05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a\": container with ID starting with 05970d3862001b76ecff0476139ea8e4703b385c04b51907fc668e84f544535a not found: ID does not exist" Jul 06 22:41:32.960257 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.960238 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68"] Jul 06 22:41:32.964132 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:32.964110 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-gnd68"] Jul 06 22:41:34.810270 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:34.810238 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" path="/var/lib/kubelet/pods/4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15/volumes" Jul 06 22:41:54.975203 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:41:54.975073 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:42:30.513415 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513378 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9"] Jul 06 22:42:30.513941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513742 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" containerName="manager" Jul 06 22:42:30.513941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513753 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" containerName="manager" Jul 06 22:42:30.513941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513770 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7411982-5615-430e-83cb-79299302ebde" containerName="authorino" Jul 06 22:42:30.513941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513775 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7411982-5615-430e-83cb-79299302ebde" containerName="authorino" Jul 06 22:42:30.513941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513852 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4971d3fe-bbfd-4ef3-a86e-f4a2e0042f15" containerName="manager" Jul 06 22:42:30.513941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.513861 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7411982-5615-430e-83cb-79299302ebde" containerName="authorino" Jul 06 22:42:30.516692 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.516676 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.519404 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.519380 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8cgv8\"" Jul 06 22:42:30.527777 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.527754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9"] Jul 06 22:42:30.586643 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.586587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhtj\" (UniqueName: \"kubernetes.io/projected/239e30b9-59f5-4e4f-808b-1c33804b5dc4-kube-api-access-kjhtj\") pod \"kuadrant-operator-controller-manager-86878c769f-bmnx9\" (UID: \"239e30b9-59f5-4e4f-808b-1c33804b5dc4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.586868 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.586727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239e30b9-59f5-4e4f-808b-1c33804b5dc4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bmnx9\" (UID: \"239e30b9-59f5-4e4f-808b-1c33804b5dc4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.688023 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.687981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhtj\" (UniqueName: \"kubernetes.io/projected/239e30b9-59f5-4e4f-808b-1c33804b5dc4-kube-api-access-kjhtj\") pod \"kuadrant-operator-controller-manager-86878c769f-bmnx9\" (UID: \"239e30b9-59f5-4e4f-808b-1c33804b5dc4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.688235 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.688109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239e30b9-59f5-4e4f-808b-1c33804b5dc4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bmnx9\" (UID: \"239e30b9-59f5-4e4f-808b-1c33804b5dc4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.688551 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.688528 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/239e30b9-59f5-4e4f-808b-1c33804b5dc4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bmnx9\" (UID: \"239e30b9-59f5-4e4f-808b-1c33804b5dc4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.696863 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.696828 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhtj\" (UniqueName: \"kubernetes.io/projected/239e30b9-59f5-4e4f-808b-1c33804b5dc4-kube-api-access-kjhtj\") pod \"kuadrant-operator-controller-manager-86878c769f-bmnx9\" (UID: \"239e30b9-59f5-4e4f-808b-1c33804b5dc4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.827749 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.827667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:30.967305 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:30.967277 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9"] Jul 06 22:42:30.970122 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:42:30.970091 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239e30b9_59f5_4e4f_808b_1c33804b5dc4.slice/crio-cceb5acbd67d0322112e90fd9fce9b3e94cfbca81ada30f718bd5070a9f62604 WatchSource:0}: Error finding container cceb5acbd67d0322112e90fd9fce9b3e94cfbca81ada30f718bd5070a9f62604: Status 404 returned error can't find the container with id cceb5acbd67d0322112e90fd9fce9b3e94cfbca81ada30f718bd5070a9f62604 Jul 06 22:42:31.224064 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:31.224026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" event={"ID":"239e30b9-59f5-4e4f-808b-1c33804b5dc4","Type":"ContainerStarted","Data":"5bf430575872dca9791bc4232892a7c92e7118f26cff942ad50e252db8b981f4"} Jul 06 22:42:31.224287 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:31.224084 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" event={"ID":"239e30b9-59f5-4e4f-808b-1c33804b5dc4","Type":"ContainerStarted","Data":"cceb5acbd67d0322112e90fd9fce9b3e94cfbca81ada30f718bd5070a9f62604"} Jul 06 22:42:31.224287 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:31.224154 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:42:31.244488 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:31.244412 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" podStartSLOduration=1.244397079 podStartE2EDuration="1.244397079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:42:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:42:31.24300131 +0000 UTC m=+1112.979403298" watchObservedRunningTime="2026-07-06 22:42:31.244397079 +0000 UTC m=+1112.980799066" Jul 06 22:42:33.989203 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:33.989167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:42:42.230923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:42:42.230885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bmnx9" Jul 06 22:43:45.959318 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:43:45.959283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:43:57.214982 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:43:57.214942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:43:58.513735 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:43:58.513698 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:43:58.739928 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:43:58.739825 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:43:58.749186 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:43:58.749162 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:47:29.020983 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:47:29.020948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:18.195798 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:18.195707 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:27.737800 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:27.737747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:35.989199 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:35.989150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:41.098552 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:41.098503 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:43.910134 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:43.910097 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:44.057168 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:44.057122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:44.393727 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:44.393690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:50.063564 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.063525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:50.164297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.164264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:50.248366 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.248334 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc"] Jul 06 22:48:50.251870 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.251847 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.254676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.254656 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c265d48a-openshift-default-dockercfg-lk7tk\"" Jul 06 22:48:50.273198 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.273174 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc"] Jul 06 22:48:50.275049 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.275024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:50.342481 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-envoy\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342643 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdg7c\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-kube-api-access-gdg7c\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342643 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-data\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342643 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istiod-ca-cert\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342643 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342610 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-credential-socket\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342643 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-podinfo\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342835 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-token\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342835 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-certs\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.342835 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.342792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-socket\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.375978 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.375939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:50.443622 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-certs\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-socket\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-envoy\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdg7c\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-kube-api-access-gdg7c\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-data\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istiod-ca-cert\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-credential-socket\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443778 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-podinfo\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.443829 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.443803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-token\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.444272 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.444144 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-certs\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.444272 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.444179 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-socket\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.444388 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.444350 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-data\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.444469 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.444381 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-credential-socket\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.444580 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.444559 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istiod-ca-cert\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.446071 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.446054 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-envoy\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.446355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.446339 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-podinfo\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.451142 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.451115 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-token\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.451288 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.451270 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdg7c\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-kube-api-access-gdg7c\") pod \"e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.562518 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.562484 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:50.705275 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.705251 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc"] Jul 06 22:48:50.707068 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:48:50.707042 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d6bcf0_c67e_485b_a8f3_56a4a6c17a77.slice/crio-19f184b29038586ca73036ed091673bbfd0b871ae6474357b8577c24a55a90e5 WatchSource:0}: Error finding container 19f184b29038586ca73036ed091673bbfd0b871ae6474357b8577c24a55a90e5: Status 404 returned error can't find the container with id 19f184b29038586ca73036ed091673bbfd0b871ae6474357b8577c24a55a90e5 Jul 06 22:48:50.709344 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.709314 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:48:50.709406 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.709379 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:48:50.709458 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:50.709406 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:48:51.077821 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.077720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" event={"ID":"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77","Type":"ContainerStarted","Data":"762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8"} Jul 06 22:48:51.077821 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.077790 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" event={"ID":"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77","Type":"ContainerStarted","Data":"19f184b29038586ca73036ed091673bbfd0b871ae6474357b8577c24a55a90e5"} Jul 06 22:48:51.097886 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.097835 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" podStartSLOduration=1.09781819 podStartE2EDuration="1.09781819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:48:51.096409682 +0000 UTC m=+1492.832811693" watchObservedRunningTime="2026-07-06 22:48:51.09781819 +0000 UTC m=+1492.834220176" Jul 06 22:48:51.366719 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.366685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:51.456092 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.456060 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:51.519122 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.519088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:51.563387 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.563339 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:51.569064 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.569037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:51.914776 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:51.914737 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:52.017910 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:52.017866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:52.083315 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:52.083278 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:52.084395 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:52.084378 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:48:52.111656 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:52.109353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:53.766719 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:53.766687 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:56.444917 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:56.444846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:56.621445 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:56.621392 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc"] Jul 06 22:48:56.621635 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:56.621609 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" podUID="f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" containerName="istio-proxy" containerID="cri-o://762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8" gracePeriod=30 Jul 06 22:48:56.637695 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:56.637662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:56.730863 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:56.730782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:56.793533 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:56.793492 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:48:58.777541 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:58.772416 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:48:58.784398 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:58.784366 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:48:58.893046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:48:58.893009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:00.804089 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:00.804047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:02.661546 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:02.661515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:02.765738 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:02.765701 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:02.876701 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:02.876669 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:02.976154 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:02.976127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:02.986822 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:02.986791 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:49:03.061785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.061737 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-token\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.061785 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.061787 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istiod-ca-cert\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.061817 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-credential-socket\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.061847 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-socket\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.061923 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-podinfo\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.061948 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-data\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-envoy\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062046 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062038 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-certs\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062092 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdg7c\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-kube-api-access-gdg7c\") pod \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\" (UID: \"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77\") " Jul 06 22:49:03.062333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062160 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:03.062333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062183 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:03.062333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062280 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:03.062572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062370 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:03.062572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062406 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istiod-ca-cert\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.062572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062461 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-credential-socket\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.062572 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062477 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-socket\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.062928 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.062906 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:03.064577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.064546 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:03.064693 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.064608 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-kube-api-access-gdg7c" (OuterVolumeSpecName: "kube-api-access-gdg7c") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "kube-api-access-gdg7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:03.064693 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.064633 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:03.064693 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.064634 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" (UID: "f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:49:03.140486 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.140449 2564 generic.go:356] "Generic (PLEG): container finished" podID="f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" containerID="762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8" exitCode=0 Jul 06 22:49:03.140676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.140491 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" Jul 06 22:49:03.140676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.140504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" event={"ID":"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77","Type":"ContainerDied","Data":"762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8"} Jul 06 22:49:03.140676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.140548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc" event={"ID":"f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77","Type":"ContainerDied","Data":"19f184b29038586ca73036ed091673bbfd0b871ae6474357b8577c24a55a90e5"} Jul 06 22:49:03.140676 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.140576 2564 scope.go:117] "RemoveContainer" containerID="762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8" Jul 06 22:49:03.150154 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.150137 2564 scope.go:117] "RemoveContainer" containerID="762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8" Jul 06 22:49:03.150416 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:49:03.150395 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8\": container with ID starting with 762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8 not found: ID does not exist" containerID="762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8" Jul 06 22:49:03.150596 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.150444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8"} err="failed to get container status \"762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8\": rpc error: code = NotFound desc = could not find container \"762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8\": container with ID starting with 762edb0631c4d6c46b0ecda4179554f834d24c7fa4fb8fc5645a0d1cd75c2ea8 not found: ID does not exist" Jul 06 22:49:03.163225 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.163202 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-token\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.163225 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.163224 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-podinfo\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.163353 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.163233 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-data\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.163353 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.163240 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-istio-envoy\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.163353 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.163248 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-workload-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.163353 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.163257 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdg7c\" (UniqueName: \"kubernetes.io/projected/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77-kube-api-access-gdg7c\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:03.164153 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.164134 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc"] Jul 06 22:49:03.168090 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.168069 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c265d48a-openshift-default-65fccd5b89-g4gfc"] Jul 06 22:49:03.957701 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:03.957668 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.040740 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:04.040697 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.115992 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:04.115943 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.345651 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:04.345597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.511137 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:04.511071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.611102 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:04.611020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:04.810583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:04.810551 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" path="/var/lib/kubelet/pods/f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77/volumes" Jul 06 22:49:06.295436 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:06.295385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:06.908593 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:06.908547 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:06.962791 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:06.962729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:09.080038 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:09.080004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:09.482251 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:09.482094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:15.314936 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:15.314892 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:15.435258 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:15.435219 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:17.483006 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:17.482976 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:21.227152 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.227009 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:21.327535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.327498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:21.410888 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.410856 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74"] Jul 06 22:49:21.411270 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.411257 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" containerName="istio-proxy" Jul 06 22:49:21.411327 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.411273 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" containerName="istio-proxy" Jul 06 22:49:21.411362 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.411346 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0d6bcf0-c67e-485b-a8f3-56a4a6c17a77" containerName="istio-proxy" Jul 06 22:49:21.414256 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.414235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.416931 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.416906 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-08e547ba-openshift-default-dockercfg-tnbjp\"" Jul 06 22:49:21.429390 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.429360 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74"] Jul 06 22:49:21.443295 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.443255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:21.503215 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.503141 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:21.532035 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-envoy\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532244 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532075 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846a0a2c-f214-47e0-842c-fbd37347ffa1-istiod-ca-cert\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532244 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532143 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-credential-socket\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532244 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-podinfo\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532244 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-socket\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532244 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-data\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532491 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-certs\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532491 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ztn9\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-kube-api-access-5ztn9\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.532491 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.532323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-token\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.633568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-credential-socket\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.633568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-podinfo\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.633837 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-socket\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.633837 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633712 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-data\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.633837 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-certs\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.633837 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ztn9\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-kube-api-access-5ztn9\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634036 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633838 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-token\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634036 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-envoy\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634036 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633988 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846a0a2c-f214-47e0-842c-fbd37347ffa1-istiod-ca-cert\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634036 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.634022 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-credential-socket\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634229 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.633987 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-socket\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634229 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.634059 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-data\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634374 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.634355 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-certs\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.634634 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.634618 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846a0a2c-f214-47e0-842c-fbd37347ffa1-istiod-ca-cert\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.636090 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.636067 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-envoy\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.636182 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.636169 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-podinfo\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.643671 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.643650 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-token\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.644061 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.644037 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ztn9\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-kube-api-access-5ztn9\") pod \"e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.725834 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.725789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:21.866505 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.866481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74"] Jul 06 22:49:21.868728 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:49:21.868702 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846a0a2c_f214_47e0_842c_fbd37347ffa1.slice/crio-eb63f377a050a04fef0bcdec053696d51d4c7f3c0e5e7da495459a75a5772128 WatchSource:0}: Error finding container eb63f377a050a04fef0bcdec053696d51d4c7f3c0e5e7da495459a75a5772128: Status 404 returned error can't find the container with id eb63f377a050a04fef0bcdec053696d51d4c7f3c0e5e7da495459a75a5772128 Jul 06 22:49:21.871035 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.871006 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:49:21.871103 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.871069 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:49:21.871141 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:21.871128 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236236Ki","pods":"250"} Jul 06 22:49:22.235714 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.235681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" event={"ID":"846a0a2c-f214-47e0-842c-fbd37347ffa1","Type":"ContainerStarted","Data":"4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757"} Jul 06 22:49:22.236148 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.235736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" event={"ID":"846a0a2c-f214-47e0-842c-fbd37347ffa1","Type":"ContainerStarted","Data":"eb63f377a050a04fef0bcdec053696d51d4c7f3c0e5e7da495459a75a5772128"} Jul 06 22:49:22.256374 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.256324 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" podStartSLOduration=1.256307162 podStartE2EDuration="1.256307162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:49:22.254957855 +0000 UTC m=+1523.991359842" watchObservedRunningTime="2026-07-06 22:49:22.256307162 +0000 UTC m=+1523.992709148" Jul 06 22:49:22.650603 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.650570 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:22.726967 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.726927 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:22.732090 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.732064 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:22.757946 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.757912 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:22.851276 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:22.851241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:23.241004 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:23.240973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:23.242001 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:23.241983 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:23.325166 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:23.325138 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:23.462013 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:23.461977 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:24.560008 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:24.559973 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:27.099142 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:27.099102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:27.232274 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:27.232238 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74"] Jul 06 22:49:27.232492 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:27.232449 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" podUID="846a0a2c-f214-47e0-842c-fbd37347ffa1" containerName="istio-proxy" containerID="cri-o://4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757" gracePeriod=30 Jul 06 22:49:27.249285 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:27.249250 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:27.370573 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:27.370480 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:27.436167 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:27.436136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:28.970613 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:28.970577 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:32.496091 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.496030 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:32.531159 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531124 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-envoy\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531159 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531161 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-token\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531210 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-credential-socket\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531251 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-podinfo\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531284 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-socket\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531319 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-certs\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531386 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531343 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ztn9\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-kube-api-access-5ztn9\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531662 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531397 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-data\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531662 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531468 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846a0a2c-f214-47e0-842c-fbd37347ffa1-istiod-ca-cert\") pod \"846a0a2c-f214-47e0-842c-fbd37347ffa1\" (UID: \"846a0a2c-f214-47e0-842c-fbd37347ffa1\") " Jul 06 22:49:32.531662 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531547 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:32.531840 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531805 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:32.531905 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531845 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-credential-socket\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.531905 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.531865 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/846a0a2c-f214-47e0-842c-fbd37347ffa1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 22:49:32.532060 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.532033 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:32.532193 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.532082 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-data" (OuterVolumeSpecName: "istio-data") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:32.533517 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.533494 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-token" (OuterVolumeSpecName: "istio-token") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:32.534029 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.534011 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 22:49:32.534114 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.534035 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:49:32.534114 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.534044 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-kube-api-access-5ztn9" (OuterVolumeSpecName: "kube-api-access-5ztn9") pod "846a0a2c-f214-47e0-842c-fbd37347ffa1" (UID: "846a0a2c-f214-47e0-842c-fbd37347ffa1"). InnerVolumeSpecName "kube-api-access-5ztn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:49:32.632871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632839 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/846a0a2c-f214-47e0-842c-fbd37347ffa1-istiod-ca-cert\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.632871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632867 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-envoy\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.632871 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632876 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-token\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.633111 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632884 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-podinfo\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.633111 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632892 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-socket\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.633111 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632899 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-workload-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.633111 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632909 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ztn9\" (UniqueName: \"kubernetes.io/projected/846a0a2c-f214-47e0-842c-fbd37347ffa1-kube-api-access-5ztn9\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:32.633111 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:32.632917 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/846a0a2c-f214-47e0-842c-fbd37347ffa1-istio-data\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:49:33.295552 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.295520 2564 generic.go:356] "Generic (PLEG): container finished" podID="846a0a2c-f214-47e0-842c-fbd37347ffa1" containerID="4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757" exitCode=0 Jul 06 22:49:33.295717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.295566 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" Jul 06 22:49:33.295717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.295568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" event={"ID":"846a0a2c-f214-47e0-842c-fbd37347ffa1","Type":"ContainerDied","Data":"4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757"} Jul 06 22:49:33.295717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.295682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74" event={"ID":"846a0a2c-f214-47e0-842c-fbd37347ffa1","Type":"ContainerDied","Data":"eb63f377a050a04fef0bcdec053696d51d4c7f3c0e5e7da495459a75a5772128"} Jul 06 22:49:33.295717 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.295711 2564 scope.go:117] "RemoveContainer" containerID="4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757" Jul 06 22:49:33.304438 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.304407 2564 scope.go:117] "RemoveContainer" containerID="4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757" Jul 06 22:49:33.304716 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:49:33.304698 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757\": container with ID starting with 4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757 not found: ID does not exist" containerID="4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757" Jul 06 22:49:33.304789 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.304724 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757"} err="failed to get container status \"4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757\": rpc error: code = NotFound desc = could not find container \"4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757\": container with ID starting with 4ada97a3944148f9a51cd3da43c7bd6ef1f902c7696b0bfc3650bd5681cb4757 not found: ID does not exist" Jul 06 22:49:33.314221 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.314198 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74"] Jul 06 22:49:33.318778 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.318757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:33.319059 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.319041 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-08e547ba-openshift-default-5cf4b8df99-wsm74"] Jul 06 22:49:33.388980 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.388945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:33.870020 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:33.869984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:34.099867 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:34.099828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:34.810576 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:34.810542 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="846a0a2c-f214-47e0-842c-fbd37347ffa1" path="/var/lib/kubelet/pods/846a0a2c-f214-47e0-842c-fbd37347ffa1/volumes" Jul 06 22:49:35.416746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:35.416716 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:41.363547 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:41.363511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:41.433273 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:41.433241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:41.509806 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:41.509773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:41.608109 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:41.608067 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:41.708488 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:41.708378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:49:42.067419 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:49:42.067312 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:07.969683 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:07.969646 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.791629 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:13.791592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.853394 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:13.853355 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:13.937739 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:13.937705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:14.035984 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:14.035939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:29.873497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:29.873459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:35.845722 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:35.845685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:35.903016 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:35.902985 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:35.978053 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:35.978013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:36.079724 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:36.079689 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:46.557483 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:46.557445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:47.324654 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:47.324620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.095441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:53.095327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.209722 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:53.209686 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.299639 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:53.299603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:53.400801 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:53.400713 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.122737 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:54.122703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.221812 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:54.221781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:54.322869 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:54.322813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:50:55.908416 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:50:55.908381 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:01.763654 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:01.763617 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:01.863574 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:01.863543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:02.424809 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:02.424779 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:08.686377 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:08.686340 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:08.757176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:08.757144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:08.925215 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:08.925178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:09.237989 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:09.237956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:10.027658 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:10.027623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:11.882272 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:11.882237 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:11.979203 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:11.979170 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:12.077600 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:12.077569 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:12.351617 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:12.351583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:12.452442 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:12.452390 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:12.550400 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:12.550371 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:16.367070 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:16.367036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:17.016972 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:17.016942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:17.116797 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:17.116764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:17.427798 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:17.427764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:23.343774 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:23.343736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:23.444099 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:23.444063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:23.822922 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:23.822836 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:24.136332 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:24.136297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:24.197239 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:24.197204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:30.292309 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:30.292276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:30.394467 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:30.394438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:30.550577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:30.550475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:30.798039 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:30.798004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:30.922454 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:30.922410 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:31.024908 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:31.024879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:31.121219 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:31.121190 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:37.106558 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:37.106525 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:37.208471 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:37.208412 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:37.306497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:37.306467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:37.448826 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:37.448747 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:43.492304 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:43.492267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:43.564087 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:43.564055 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:50.178112 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:50.178069 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:50.483213 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:50.483140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:50.846101 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:50.846063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:52.899333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:52.899296 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:52.998552 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:52.998506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:53.100905 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:53.100879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:53.386233 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:53.386198 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:53.566733 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:53.566702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:53.712478 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:53.712377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:51:57.401647 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:51:57.401614 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:05.545152 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:05.545118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:05.889396 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:05.889361 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:05.987837 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:05.987797 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:09.716887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.716854 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs"] Jul 06 22:52:09.717277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.717203 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846a0a2c-f214-47e0-842c-fbd37347ffa1" containerName="istio-proxy" Jul 06 22:52:09.717277 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.717214 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="846a0a2c-f214-47e0-842c-fbd37347ffa1" containerName="istio-proxy" Jul 06 22:52:09.717352 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.717282 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="846a0a2c-f214-47e0-842c-fbd37347ffa1" containerName="istio-proxy" Jul 06 22:52:09.720408 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.720387 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.723690 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.723670 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fed91589\"/\"kube-root-ca.crt\"" Jul 06 22:52:09.724793 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.724773 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fed91589\"/\"e2e-lifecycle-model-fed91589-kserve-self-signed-certs\"" Jul 06 22:52:09.724919 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.724820 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-fed91589\"/\"openshift-service-ca.crt\"" Jul 06 22:52:09.724919 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.724850 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-fed91589\"/\"default-dockercfg-5ttxx\"" Jul 06 22:52:09.731836 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.731816 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs"] Jul 06 22:52:09.740016 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.739994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:09.860312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.860215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xdgr\" (UniqueName: \"kubernetes.io/projected/35820772-d927-4b2e-90bc-4e20f24fc9c0-kube-api-access-5xdgr\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.860312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.860263 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-dshm\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.860312 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.860299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-tmp-dir\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.860664 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.860438 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-home\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.860664 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.860497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-model-cache\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.860664 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.860523 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35820772-d927-4b2e-90bc-4e20f24fc9c0-tls-certs\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-model-cache\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35820772-d927-4b2e-90bc-4e20f24fc9c0-tls-certs\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961461 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961240 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xdgr\" (UniqueName: \"kubernetes.io/projected/35820772-d927-4b2e-90bc-4e20f24fc9c0-kube-api-access-5xdgr\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961461 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961265 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-dshm\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961461 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-tmp-dir\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961461 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-home\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961691 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961588 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-model-cache\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961691 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961666 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-tmp-dir\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.961770 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.961691 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-home\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.963596 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.963574 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-dshm\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.963784 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.963764 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35820772-d927-4b2e-90bc-4e20f24fc9c0-tls-certs\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:09.968722 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:09.968669 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xdgr\" (UniqueName: \"kubernetes.io/projected/35820772-d927-4b2e-90bc-4e20f24fc9c0-kube-api-access-5xdgr\") pod \"e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:10.031568 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:10.031529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:10.175531 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:10.175499 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs"] Jul 06 22:52:10.177980 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:52:10.177940 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35820772_d927_4b2e_90bc_4e20f24fc9c0.slice/crio-cbdcb62a9cff44758d0e39a97382222aeb1555065eff802511382789f24063e0 WatchSource:0}: Error finding container cbdcb62a9cff44758d0e39a97382222aeb1555065eff802511382789f24063e0: Status 404 returned error can't find the container with id cbdcb62a9cff44758d0e39a97382222aeb1555065eff802511382789f24063e0 Jul 06 22:52:11.056488 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:11.056448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" event={"ID":"35820772-d927-4b2e-90bc-4e20f24fc9c0","Type":"ContainerStarted","Data":"34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235"} Jul 06 22:52:11.056488 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:11.056494 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" event={"ID":"35820772-d927-4b2e-90bc-4e20f24fc9c0","Type":"ContainerStarted","Data":"cbdcb62a9cff44758d0e39a97382222aeb1555065eff802511382789f24063e0"} Jul 06 22:52:11.074502 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:11.074448 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" podStartSLOduration=2.074413707 podStartE2EDuration="2.074413707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:52:11.073124381 +0000 UTC m=+1692.809526368" watchObservedRunningTime="2026-07-06 22:52:11.074413707 +0000 UTC m=+1692.810815759" Jul 06 22:52:14.314435 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:14.314399 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:14.418412 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:14.418377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:20.032310 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:20.032273 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:20.032758 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:20.032331 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:20.045331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:20.045303 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:20.112452 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:20.112404 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:29.670949 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:29.670915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:30.116414 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:30.116386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:35.786575 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:35.786538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:40.762768 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:40.762737 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs"] Jul 06 22:52:40.858032 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:40.857994 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:52:55.783722 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:55.783678 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" podUID="35820772-d927-4b2e-90bc-4e20f24fc9c0" containerName="main" containerID="cri-o://34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235" gracePeriod=45 Jul 06 22:52:56.038298 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.038232 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:56.203047 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203008 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-dshm\") pod \"35820772-d927-4b2e-90bc-4e20f24fc9c0\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " Jul 06 22:52:56.203047 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203050 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-tmp-dir\") pod \"35820772-d927-4b2e-90bc-4e20f24fc9c0\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " Jul 06 22:52:56.203314 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203089 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35820772-d927-4b2e-90bc-4e20f24fc9c0-tls-certs\") pod \"35820772-d927-4b2e-90bc-4e20f24fc9c0\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " Jul 06 22:52:56.203314 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203120 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xdgr\" (UniqueName: \"kubernetes.io/projected/35820772-d927-4b2e-90bc-4e20f24fc9c0-kube-api-access-5xdgr\") pod \"35820772-d927-4b2e-90bc-4e20f24fc9c0\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " Jul 06 22:52:56.203314 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203154 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-model-cache\") pod \"35820772-d927-4b2e-90bc-4e20f24fc9c0\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " Jul 06 22:52:56.203314 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203220 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-home\") pod \"35820772-d927-4b2e-90bc-4e20f24fc9c0\" (UID: \"35820772-d927-4b2e-90bc-4e20f24fc9c0\") " Jul 06 22:52:56.203583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203391 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "35820772-d927-4b2e-90bc-4e20f24fc9c0" (UID: "35820772-d927-4b2e-90bc-4e20f24fc9c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:56.203583 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203518 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "35820772-d927-4b2e-90bc-4e20f24fc9c0" (UID: "35820772-d927-4b2e-90bc-4e20f24fc9c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:56.203695 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203602 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:52:56.203695 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.203625 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-home" (OuterVolumeSpecName: "home") pod "35820772-d927-4b2e-90bc-4e20f24fc9c0" (UID: "35820772-d927-4b2e-90bc-4e20f24fc9c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:56.205363 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.205330 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35820772-d927-4b2e-90bc-4e20f24fc9c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "35820772-d927-4b2e-90bc-4e20f24fc9c0" (UID: "35820772-d927-4b2e-90bc-4e20f24fc9c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:52:56.205497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.205362 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-dshm" (OuterVolumeSpecName: "dshm") pod "35820772-d927-4b2e-90bc-4e20f24fc9c0" (UID: "35820772-d927-4b2e-90bc-4e20f24fc9c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:52:56.205497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.205376 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35820772-d927-4b2e-90bc-4e20f24fc9c0-kube-api-access-5xdgr" (OuterVolumeSpecName: "kube-api-access-5xdgr") pod "35820772-d927-4b2e-90bc-4e20f24fc9c0" (UID: "35820772-d927-4b2e-90bc-4e20f24fc9c0"). InnerVolumeSpecName "kube-api-access-5xdgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:52:56.277542 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.277502 2564 generic.go:356] "Generic (PLEG): container finished" podID="35820772-d927-4b2e-90bc-4e20f24fc9c0" containerID="34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235" exitCode=0 Jul 06 22:52:56.277713 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.277550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" Jul 06 22:52:56.277713 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.277550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" event={"ID":"35820772-d927-4b2e-90bc-4e20f24fc9c0","Type":"ContainerDied","Data":"34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235"} Jul 06 22:52:56.277713 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.277597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs" event={"ID":"35820772-d927-4b2e-90bc-4e20f24fc9c0","Type":"ContainerDied","Data":"cbdcb62a9cff44758d0e39a97382222aeb1555065eff802511382789f24063e0"} Jul 06 22:52:56.277713 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.277615 2564 scope.go:117] "RemoveContainer" containerID="34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235" Jul 06 22:52:56.286899 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.286882 2564 scope.go:117] "RemoveContainer" containerID="34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235" Jul 06 22:52:56.287181 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:52:56.287161 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235\": container with ID starting with 34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235 not found: ID does not exist" containerID="34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235" Jul 06 22:52:56.287232 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.287190 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235"} err="failed to get container status \"34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235\": rpc error: code = NotFound desc = could not find container \"34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235\": container with ID starting with 34fff222c70c1cce1eb97e85276fd8df02bbfef2fac5ec7ba0946f03af82c235 not found: ID does not exist" Jul 06 22:52:56.299271 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.299221 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs"] Jul 06 22:52:56.302654 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.302635 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-fed91589/e2e-lifecycle-model-fed91589-kserve-76f8b5885f-dswhs"] Jul 06 22:52:56.304805 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.304789 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:52:56.304857 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.304809 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35820772-d927-4b2e-90bc-4e20f24fc9c0-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:52:56.304857 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.304820 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xdgr\" (UniqueName: \"kubernetes.io/projected/35820772-d927-4b2e-90bc-4e20f24fc9c0-kube-api-access-5xdgr\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:52:56.304857 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.304828 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:52:56.304857 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.304836 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35820772-d927-4b2e-90bc-4e20f24fc9c0-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:52:56.810807 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:52:56.810778 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35820772-d927-4b2e-90bc-4e20f24fc9c0" path="/var/lib/kubelet/pods/35820772-d927-4b2e-90bc-4e20f24fc9c0/volumes" Jul 06 22:53:17.075328 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:17.075289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:17.963689 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:17.963650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:23.551825 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:23.551776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:23.652301 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:23.652267 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:23.830284 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:23.830252 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:24.052441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:24.052398 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:25.101486 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:25.101449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:25.190953 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:25.190906 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:25.289296 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:25.289259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:26.007642 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:26.007587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:26.091554 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:26.091518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:26.191480 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:26.191448 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:27.528778 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:27.528745 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:33.347691 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:33.347646 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:33.405849 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:33.405816 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:34.017128 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:34.017079 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:39.867300 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:39.867263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:39.932763 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:39.932728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:40.041328 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:40.041290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:50.561618 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:50.561576 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:50.916770 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:50.916733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:53:58.810994 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:58.810961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:53:58.823531 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:53:58.823507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:54:11.830584 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:11.830546 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:17.690904 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:17.690870 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:17.755623 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:17.755596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:18.024779 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:18.024685 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:18.400122 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:18.400087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:18.767293 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:18.767197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:21.653441 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:21.653394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:21.754776 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:21.754739 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:21.969897 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:21.969807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:22.057122 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:22.057092 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:22.412520 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:22.412486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:22.513088 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:22.513052 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:26.126634 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:26.126601 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:26.429528 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:26.429445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:26.754245 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:26.754145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:26.997122 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:26.997093 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:29.678121 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:29.678085 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:29.764340 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:29.764304 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:29.840297 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:29.840264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:30.041887 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:30.041796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:30.220657 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:30.220615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:30.321271 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:30.321185 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:34.152228 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:34.152193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:43.766499 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:43.766463 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:43.866535 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:43.866501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:48.003136 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.003097 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd"] Jul 06 22:54:48.003712 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.003694 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35820772-d927-4b2e-90bc-4e20f24fc9c0" containerName="main" Jul 06 22:54:48.003796 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.003715 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="35820772-d927-4b2e-90bc-4e20f24fc9c0" containerName="main" Jul 06 22:54:48.003854 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.003825 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="35820772-d927-4b2e-90bc-4e20f24fc9c0" containerName="main" Jul 06 22:54:48.007146 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.007124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.010923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.010871 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b673d1\"/\"openshift-service-ca.crt\"" Jul 06 22:54:48.010923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.010887 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b673d1\"/\"default-dockercfg-hndlt\"" Jul 06 22:54:48.010923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.010921 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b673d1\"/\"kube-root-ca.crt\"" Jul 06 22:54:48.011342 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.010923 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b673d1\"/\"test-model-b673d1-kserve-self-signed-certs\"" Jul 06 22:54:48.013666 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.013553 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd"] Jul 06 22:54:48.024882 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.024848 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:48.142615 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.142578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-home\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.142799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.142653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj5mt\" (UniqueName: \"kubernetes.io/projected/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-kube-api-access-wj5mt\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.142799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.142688 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tmp-dir\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.142799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.142712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tls-certs\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.142935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.142791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-model-cache\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.142935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.142823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-dshm\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243369 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-model-cache\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243369 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-dshm\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243614 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-home\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243614 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj5mt\" (UniqueName: \"kubernetes.io/projected/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-kube-api-access-wj5mt\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243614 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tmp-dir\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243614 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243590 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tls-certs\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243841 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243815 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-model-cache\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243904 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243889 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-home\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.243983 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.243962 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tmp-dir\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.245799 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.245775 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-dshm\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.245991 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.245975 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tls-certs\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.251219 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.251198 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj5mt\" (UniqueName: \"kubernetes.io/projected/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-kube-api-access-wj5mt\") pod \"test-model-b673d1-kserve-6b9c456747-s6dgd\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.318313 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.318222 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:48.454592 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.454562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd"] Jul 06 22:54:48.456484 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:54:48.456456 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeea90d7d_7deb_4bb7_afa2_69ddc3e0ec73.slice/crio-1877c4f39aa520b5f6b19a404dc88d69963d74ce0d2b33f16ce73e10cb464c9e WatchSource:0}: Error finding container 1877c4f39aa520b5f6b19a404dc88d69963d74ce0d2b33f16ce73e10cb464c9e: Status 404 returned error can't find the container with id 1877c4f39aa520b5f6b19a404dc88d69963d74ce0d2b33f16ce73e10cb464c9e Jul 06 22:54:48.810834 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.810797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" event={"ID":"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73","Type":"ContainerStarted","Data":"c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7"} Jul 06 22:54:48.811026 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.810844 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" event={"ID":"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73","Type":"ContainerStarted","Data":"1877c4f39aa520b5f6b19a404dc88d69963d74ce0d2b33f16ce73e10cb464c9e"} Jul 06 22:54:48.826602 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:48.826549 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" podStartSLOduration=1.826531701 podStartE2EDuration="1.826531701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:54:48.825749884 +0000 UTC m=+1850.562151869" watchObservedRunningTime="2026-07-06 22:54:48.826531701 +0000 UTC m=+1850.562933724" Jul 06 22:54:51.747169 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:51.747136 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:51.847563 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:51.847527 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:54:58.318496 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:58.318454 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:58.319015 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:58.318522 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:58.331577 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:58.331552 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:54:58.870552 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:54:58.870520 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:55:10.615983 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.615942 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx"] Jul 06 22:55:10.619754 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.619730 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.622333 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.622311 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f49549\"/\"openshift-service-ca.crt\"" Jul 06 22:55:10.623370 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.623333 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f49549\"/\"default-dockercfg-5srng\"" Jul 06 22:55:10.623528 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.623384 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f49549\"/\"kube-root-ca.crt\"" Jul 06 22:55:10.623528 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.623462 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f49549\"/\"test-model-f49549-kserve-self-signed-certs\"" Jul 06 22:55:10.626971 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.626946 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx"] Jul 06 22:55:10.644095 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.644040 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:10.760056 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.760018 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-dshm\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.760056 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.760056 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tmp-dir\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.760319 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.760091 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-model-cache\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.760319 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.760199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-home\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.760319 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.760234 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tls-certs\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.760319 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.760290 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b25w\" (UniqueName: \"kubernetes.io/projected/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-kube-api-access-5b25w\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861205 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-home\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861205 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861209 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tls-certs\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861453 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b25w\" (UniqueName: \"kubernetes.io/projected/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-kube-api-access-5b25w\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861453 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-dshm\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861453 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861377 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tmp-dir\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861453 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861418 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-model-cache\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861659 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861585 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-home\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861768 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861749 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-model-cache\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.861842 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.861801 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tmp-dir\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.863658 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.863640 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-dshm\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.863842 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.863826 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tls-certs\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.869804 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.869745 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b25w\" (UniqueName: \"kubernetes.io/projected/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-kube-api-access-5b25w\") pod \"test-model-f49549-kserve-c7f7f79bc-khlkx\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:10.930053 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:10.930019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:11.067200 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:11.067163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx"] Jul 06 22:55:11.071523 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:55:11.071460 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b3b6b9c_b723_4fbf_9979_a3637ab722fe.slice/crio-66b7110f358e047fa463baa414bba2c777387df8124db6bc459cacb584bea846 WatchSource:0}: Error finding container 66b7110f358e047fa463baa414bba2c777387df8124db6bc459cacb584bea846: Status 404 returned error can't find the container with id 66b7110f358e047fa463baa414bba2c777387df8124db6bc459cacb584bea846 Jul 06 22:55:11.924687 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:11.924653 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" event={"ID":"6b3b6b9c-b723-4fbf-9979-a3637ab722fe","Type":"ContainerStarted","Data":"e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227"} Jul 06 22:55:11.925102 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:11.924695 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" event={"ID":"6b3b6b9c-b723-4fbf-9979-a3637ab722fe","Type":"ContainerStarted","Data":"66b7110f358e047fa463baa414bba2c777387df8124db6bc459cacb584bea846"} Jul 06 22:55:11.941641 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:11.941589 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" podStartSLOduration=1.9415727280000001 podStartE2EDuration="1.941572728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:55:11.94029432 +0000 UTC m=+1873.676696308" watchObservedRunningTime="2026-07-06 22:55:11.941572728 +0000 UTC m=+1873.677974715" Jul 06 22:55:20.931077 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:20.931035 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:20.931528 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:20.931090 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:20.944092 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:20.944062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:20.983454 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:20.983402 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:27.325540 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:27.325509 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd"] Jul 06 22:55:27.346269 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:27.346231 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:28.259376 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:28.259339 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx"] Jul 06 22:55:28.279603 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:28.279571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:28.423217 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:28.423187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:28.813113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:28.813082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:28.946632 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:28.946596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:29.047279 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:29.047248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:29.177876 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:29.177847 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:33.896327 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:33.896294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:55:42.346410 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.346368 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" podUID="eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" containerName="main" containerID="cri-o://c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7" gracePeriod=45 Jul 06 22:55:42.613086 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.613013 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:55:42.751935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.751895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-model-cache\") pod \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " Jul 06 22:55:42.752131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.751990 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj5mt\" (UniqueName: \"kubernetes.io/projected/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-kube-api-access-wj5mt\") pod \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " Jul 06 22:55:42.752131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752023 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-dshm\") pod \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " Jul 06 22:55:42.752131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752049 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tls-certs\") pod \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " Jul 06 22:55:42.752131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tmp-dir\") pod \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " Jul 06 22:55:42.752348 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752215 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-home\") pod \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\" (UID: \"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73\") " Jul 06 22:55:42.752411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752216 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-model-cache" (OuterVolumeSpecName: "model-cache") pod "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" (UID: "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:42.752411 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752313 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" (UID: "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:42.752566 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752456 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-home" (OuterVolumeSpecName: "home") pod "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" (UID: "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:42.752619 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752568 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:42.752619 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752582 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:42.752619 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.752590 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:42.754389 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.754362 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-dshm" (OuterVolumeSpecName: "dshm") pod "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" (UID: "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:42.754519 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.754440 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" (UID: "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:55:42.754519 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.754464 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-kube-api-access-wj5mt" (OuterVolumeSpecName: "kube-api-access-wj5mt") pod "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" (UID: "eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73"). InnerVolumeSpecName "kube-api-access-wj5mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:55:42.853393 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.853353 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj5mt\" (UniqueName: \"kubernetes.io/projected/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-kube-api-access-wj5mt\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:42.853393 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.853387 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:42.853393 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:42.853397 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:43.072881 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.072784 2564 generic.go:356] "Generic (PLEG): container finished" podID="eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" containerID="c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7" exitCode=0 Jul 06 22:55:43.072881 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.072836 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" Jul 06 22:55:43.073113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.072873 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" event={"ID":"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73","Type":"ContainerDied","Data":"c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7"} Jul 06 22:55:43.073113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.072919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd" event={"ID":"eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73","Type":"ContainerDied","Data":"1877c4f39aa520b5f6b19a404dc88d69963d74ce0d2b33f16ce73e10cb464c9e"} Jul 06 22:55:43.073113 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.072937 2564 scope.go:117] "RemoveContainer" containerID="c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7" Jul 06 22:55:43.082007 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.081987 2564 scope.go:117] "RemoveContainer" containerID="c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7" Jul 06 22:55:43.082351 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:55:43.082321 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7\": container with ID starting with c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7 not found: ID does not exist" containerID="c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7" Jul 06 22:55:43.082470 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.082366 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7"} err="failed to get container status \"c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7\": rpc error: code = NotFound desc = could not find container \"c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7\": container with ID starting with c03ed052bd2b366a888bf6fe7c6af9d28c406551ac99c08e85e8d229c39f1ec7 not found: ID does not exist" Jul 06 22:55:43.089335 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.089308 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd"] Jul 06 22:55:43.091022 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.091003 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b673d1/test-model-b673d1-kserve-6b9c456747-s6dgd"] Jul 06 22:55:43.285092 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.285031 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" podUID="6b3b6b9c-b723-4fbf-9979-a3637ab722fe" containerName="main" containerID="cri-o://e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227" gracePeriod=45 Jul 06 22:55:43.545504 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.545471 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:43.662176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662076 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-dshm\") pod \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " Jul 06 22:55:43.662176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662114 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tls-certs\") pod \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " Jul 06 22:55:43.662176 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-home\") pod \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " Jul 06 22:55:43.662512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662169 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tmp-dir\") pod \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " Jul 06 22:55:43.662512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662264 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b25w\" (UniqueName: \"kubernetes.io/projected/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-kube-api-access-5b25w\") pod \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " Jul 06 22:55:43.662512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-model-cache\") pod \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\" (UID: \"6b3b6b9c-b723-4fbf-9979-a3637ab722fe\") " Jul 06 22:55:43.662512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662459 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b3b6b9c-b723-4fbf-9979-a3637ab722fe" (UID: "6b3b6b9c-b723-4fbf-9979-a3637ab722fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:43.662512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662464 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-home" (OuterVolumeSpecName: "home") pod "6b3b6b9c-b723-4fbf-9979-a3637ab722fe" (UID: "6b3b6b9c-b723-4fbf-9979-a3637ab722fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:43.662780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662602 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b3b6b9c-b723-4fbf-9979-a3637ab722fe" (UID: "6b3b6b9c-b723-4fbf-9979-a3637ab722fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:43.662780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662689 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:43.662780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662700 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:43.662780 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.662709 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:43.664466 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.664418 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-dshm" (OuterVolumeSpecName: "dshm") pod "6b3b6b9c-b723-4fbf-9979-a3637ab722fe" (UID: "6b3b6b9c-b723-4fbf-9979-a3637ab722fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 22:55:43.664580 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.664481 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b3b6b9c-b723-4fbf-9979-a3637ab722fe" (UID: "6b3b6b9c-b723-4fbf-9979-a3637ab722fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 22:55:43.664580 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.664508 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-kube-api-access-5b25w" (OuterVolumeSpecName: "kube-api-access-5b25w") pod "6b3b6b9c-b723-4fbf-9979-a3637ab722fe" (UID: "6b3b6b9c-b723-4fbf-9979-a3637ab722fe"). InnerVolumeSpecName "kube-api-access-5b25w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 22:55:43.764050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.764009 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:43.764050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.764044 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:43.764050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:43.764056 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b25w\" (UniqueName: \"kubernetes.io/projected/6b3b6b9c-b723-4fbf-9979-a3637ab722fe-kube-api-access-5b25w\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 22:55:44.079512 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.079481 2564 generic.go:356] "Generic (PLEG): container finished" podID="6b3b6b9c-b723-4fbf-9979-a3637ab722fe" containerID="e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227" exitCode=0 Jul 06 22:55:44.079706 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.079530 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" Jul 06 22:55:44.079706 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.079572 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" event={"ID":"6b3b6b9c-b723-4fbf-9979-a3637ab722fe","Type":"ContainerDied","Data":"e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227"} Jul 06 22:55:44.079706 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.079614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx" event={"ID":"6b3b6b9c-b723-4fbf-9979-a3637ab722fe","Type":"ContainerDied","Data":"66b7110f358e047fa463baa414bba2c777387df8124db6bc459cacb584bea846"} Jul 06 22:55:44.079706 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.079631 2564 scope.go:117] "RemoveContainer" containerID="e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227" Jul 06 22:55:44.095439 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.095401 2564 scope.go:117] "RemoveContainer" containerID="e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227" Jul 06 22:55:44.095742 ip-10-0-135-55 kubenswrapper[2564]: E0706 22:55:44.095723 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227\": container with ID starting with e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227 not found: ID does not exist" containerID="e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227" Jul 06 22:55:44.095802 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.095752 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227"} err="failed to get container status \"e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227\": rpc error: code = NotFound desc = could not find container \"e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227\": container with ID starting with e2066cc70d886b1793a9bd60147d28d77739eb6143107329c9aa1f7a62cf1227 not found: ID does not exist" Jul 06 22:55:44.103889 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.103863 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx"] Jul 06 22:55:44.107218 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.107196 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f49549/test-model-f49549-kserve-c7f7f79bc-khlkx"] Jul 06 22:55:44.810225 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.810185 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3b6b9c-b723-4fbf-9979-a3637ab722fe" path="/var/lib/kubelet/pods/6b3b6b9c-b723-4fbf-9979-a3637ab722fe/volumes" Jul 06 22:55:44.810673 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:55:44.810658 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" path="/var/lib/kubelet/pods/eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73/volumes" Jul 06 22:56:04.483681 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:04.483624 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:04.537117 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:04.537064 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:10.973506 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:10.973462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:11.236051 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:11.235940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:11.461342 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:11.461308 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:11.613367 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:11.613321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:12.008960 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:12.008871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:12.132475 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:12.132444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:12.220326 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:12.220283 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:12.353587 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:12.353550 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:18.018388 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:18.018353 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:18.084649 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:18.084618 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:18.283408 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:18.283306 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:18.502778 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:18.502738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:18.836378 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:18.836347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:19.113582 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:19.113474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:19.349049 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:19.349015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:22.300341 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:22.300290 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:22.469355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:22.469316 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:22.596620 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:22.596588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:22.964448 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:22.964337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:23.065108 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:23.065076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:23.165860 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:23.165830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:26.753790 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:26.753759 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:27.192019 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:27.191971 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:27.500136 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:27.500048 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:27.734477 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:27.734415 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:30.347246 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:30.347197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:30.472240 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:30.472207 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:30.595935 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:30.595900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:31.049355 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:31.049319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:31.150672 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:31.150637 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:31.251941 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:31.251902 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:34.808654 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:34.808616 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:42.828559 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:42.828523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.243125 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:43.243028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.403686 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:43.403652 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.504723 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:43.504622 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:43.651682 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:43.651651 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:49.075025 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:49.074989 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:49.172050 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:49.172011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:49.323839 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:49.323800 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:49.554709 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:49.554674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:49.707639 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:49.707603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:50.106992 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:50.106946 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:50.206915 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:50.206879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:50.345523 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:50.345490 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:56.241259 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:56.241168 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:56.335879 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:56.335843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:56.472659 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:56.472620 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:56.679299 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:56.679259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:56.999331 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:56.999234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:57.290192 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:57.290090 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:56:57.615896 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:56:57.615865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:00.506936 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:00.506904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:00.596069 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:00.596036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:00.674282 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:00.674230 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:00.884985 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:00.884950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:01.088841 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:01.088810 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:01.204131 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:01.204044 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:04.973219 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:04.973186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:05.283128 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:05.283030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:05.566254 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:05.566158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:06.002035 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:06.002005 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:06.558323 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:06.558291 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:06.663934 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:06.663903 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:06.880979 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:06.880949 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:08.527997 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:08.527955 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:08.611637 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:08.611590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:08.687909 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:08.687876 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:08.913184 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:08.913157 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:09.055226 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:09.055193 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:09.155840 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:09.155799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:13.002463 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:13.002407 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:20.500509 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500471 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9"] Jul 06 22:57:20.500990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500850 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" containerName="main" Jul 06 22:57:20.500990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500865 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" containerName="main" Jul 06 22:57:20.500990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500884 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3b6b9c-b723-4fbf-9979-a3637ab722fe" containerName="main" Jul 06 22:57:20.500990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500890 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3b6b9c-b723-4fbf-9979-a3637ab722fe" containerName="main" Jul 06 22:57:20.500990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500945 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3b6b9c-b723-4fbf-9979-a3637ab722fe" containerName="main" Jul 06 22:57:20.500990 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.500957 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eea90d7d-7deb-4bb7-afa2-69ddc3e0ec73" containerName="main" Jul 06 22:57:20.504354 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.504335 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.506923 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.506901 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"kube-root-ca.crt\"" Jul 06 22:57:20.508083 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.508063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"default-dockercfg-cl27v\"" Jul 06 22:57:20.508187 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.508111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"auth-test-model-936b5d-kserve-self-signed-certs\"" Jul 06 22:57:20.508187 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.508135 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"openshift-service-ca.crt\"" Jul 06 22:57:20.512867 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.512845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9"] Jul 06 22:57:20.602276 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.602228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:20.641849 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.641812 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-dshm\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.642065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.641894 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-home\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.642065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.641912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-model-cache\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.642065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.641938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tls-certs\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.642065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.641953 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9g7m\" (UniqueName: \"kubernetes.io/projected/e539e2e4-5a4f-4282-8628-bf10e656c7c7-kube-api-access-j9g7m\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.642065 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.642015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tmp-dir\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-home\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743497 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-model-cache\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743531 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tls-certs\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9g7m\" (UniqueName: \"kubernetes.io/projected/e539e2e4-5a4f-4282-8628-bf10e656c7c7-kube-api-access-j9g7m\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tmp-dir\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743746 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-dshm\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743966 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743869 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-home\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.743966 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.743942 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-model-cache\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.744074 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.744047 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tmp-dir\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.745836 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.745814 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-dshm\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.746095 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.746078 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tls-certs\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.751345 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.751293 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9g7m\" (UniqueName: \"kubernetes.io/projected/e539e2e4-5a4f-4282-8628-bf10e656c7c7-kube-api-access-j9g7m\") pod \"auth-test-model-936b5d-kserve-6f8d57f487-4q4r9\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.814330 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.814298 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:20.956219 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:20.956194 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9"] Jul 06 22:57:20.958457 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:57:20.958416 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode539e2e4_5a4f_4282_8628_bf10e656c7c7.slice/crio-fd47494381330baa95ca449fff1f3fd6cc7a21bec7a8e7c617e53302aea6b818 WatchSource:0}: Error finding container fd47494381330baa95ca449fff1f3fd6cc7a21bec7a8e7c617e53302aea6b818: Status 404 returned error can't find the container with id fd47494381330baa95ca449fff1f3fd6cc7a21bec7a8e7c617e53302aea6b818 Jul 06 22:57:21.559350 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:21.559313 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" event={"ID":"e539e2e4-5a4f-4282-8628-bf10e656c7c7","Type":"ContainerStarted","Data":"6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af"} Jul 06 22:57:21.559350 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:21.559354 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" event={"ID":"e539e2e4-5a4f-4282-8628-bf10e656c7c7","Type":"ContainerStarted","Data":"fd47494381330baa95ca449fff1f3fd6cc7a21bec7a8e7c617e53302aea6b818"} Jul 06 22:57:21.576686 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:21.576631 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" podStartSLOduration=1.576612914 podStartE2EDuration="1.576612914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:57:21.575638364 +0000 UTC m=+2003.312040376" watchObservedRunningTime="2026-07-06 22:57:21.576612914 +0000 UTC m=+2003.313014906" Jul 06 22:57:22.715507 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:22.715468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:22.814808 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:22.814773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:29.910958 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:29.910904 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:30.008837 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:30.008802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:57:30.814982 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:30.814952 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:30.815236 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:30.814997 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:30.827618 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:30.827590 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:57:31.618192 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:57:31.618161 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 22:58:02.611112 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.611079 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6"] Jul 06 22:58:02.614947 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.614926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.618703 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.618670 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"kube-root-ca.crt\"" Jul 06 22:58:02.618843 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.618725 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"default-dockercfg-s796x\"" Jul 06 22:58:02.618843 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.618670 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"openshift-service-ca.crt\"" Jul 06 22:58:02.618970 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.618772 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"auth-test-model-cf2b54-kserve-self-signed-certs\"" Jul 06 22:58:02.622548 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.622392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6"] Jul 06 22:58:02.653648 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.653619 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 22:58:02.733598 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.733561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-dshm\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.733598 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.733603 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6lq6\" (UniqueName: \"kubernetes.io/projected/869ba53f-531e-4be9-9513-724732594a70-kube-api-access-c6lq6\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.733833 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.733634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869ba53f-531e-4be9-9513-724732594a70-tls-certs\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.733833 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.733665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-home\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.733833 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.733699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-tmp-dir\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.733833 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.733773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-model-cache\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834510 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834475 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-dshm\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834510 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834516 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6lq6\" (UniqueName: \"kubernetes.io/projected/869ba53f-531e-4be9-9513-724732594a70-kube-api-access-c6lq6\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869ba53f-531e-4be9-9513-724732594a70-tls-certs\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-home\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-tmp-dir\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834755 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-model-cache\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.834968 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.834954 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-home\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.835056 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.835035 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-model-cache\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.835118 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.835090 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-tmp-dir\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.836951 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.836920 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-dshm\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.837156 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.837135 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869ba53f-531e-4be9-9513-724732594a70-tls-certs\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.842460 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.842439 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6lq6\" (UniqueName: \"kubernetes.io/projected/869ba53f-531e-4be9-9513-724732594a70-kube-api-access-c6lq6\") pod \"auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:02.927598 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:02.927507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:03.071230 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:03.071198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6"] Jul 06 22:58:03.073912 ip-10-0-135-55 kubenswrapper[2564]: W0706 22:58:03.073878 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod869ba53f_531e_4be9_9513_724732594a70.slice/crio-47d63b430e4621042bec43803a413d8cce826fbfcfa80075677252b603016896 WatchSource:0}: Error finding container 47d63b430e4621042bec43803a413d8cce826fbfcfa80075677252b603016896: Status 404 returned error can't find the container with id 47d63b430e4621042bec43803a413d8cce826fbfcfa80075677252b603016896 Jul 06 22:58:03.763051 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:03.763016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" event={"ID":"869ba53f-531e-4be9-9513-724732594a70","Type":"ContainerStarted","Data":"bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f"} Jul 06 22:58:03.763482 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:03.763062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" event={"ID":"869ba53f-531e-4be9-9513-724732594a70","Type":"ContainerStarted","Data":"47d63b430e4621042bec43803a413d8cce826fbfcfa80075677252b603016896"} Jul 06 22:58:03.779516 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:03.779457 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" podStartSLOduration=1.779434304 podStartE2EDuration="1.779434304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 22:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 22:58:03.778004366 +0000 UTC m=+2045.514406355" watchObservedRunningTime="2026-07-06 22:58:03.779434304 +0000 UTC m=+2045.515836285" Jul 06 22:58:12.928227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:12.928196 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:12.928726 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:12.928245 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:12.941330 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:12.941305 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:13.828187 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:13.828146 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 22:58:58.840038 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:58.840010 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:58:58.849781 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:58:58.849755 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 22:59:35.495227 ip-10-0-135-55 kubenswrapper[2564]: I0706 22:59:35.495187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:00.318347 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:00.318255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:07.091909 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:07.091875 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9"] Jul 06 23:00:07.110254 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:07.110209 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:07.918857 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:07.918820 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6"] Jul 06 23:00:07.941112 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:07.941076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:20.241601 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:20.241562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:22.114249 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.114181 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" podUID="e539e2e4-5a4f-4282-8628-bf10e656c7c7" containerName="main" containerID="cri-o://6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af" gracePeriod=45 Jul 06 23:00:22.386749 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.386684 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 23:00:22.438384 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.438351 2564 generic.go:356] "Generic (PLEG): container finished" podID="e539e2e4-5a4f-4282-8628-bf10e656c7c7" containerID="6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af" exitCode=0 Jul 06 23:00:22.438582 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.438394 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" Jul 06 23:00:22.438582 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.438447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" event={"ID":"e539e2e4-5a4f-4282-8628-bf10e656c7c7","Type":"ContainerDied","Data":"6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af"} Jul 06 23:00:22.438582 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.438488 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9" event={"ID":"e539e2e4-5a4f-4282-8628-bf10e656c7c7","Type":"ContainerDied","Data":"fd47494381330baa95ca449fff1f3fd6cc7a21bec7a8e7c617e53302aea6b818"} Jul 06 23:00:22.438582 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.438505 2564 scope.go:117] "RemoveContainer" containerID="6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af" Jul 06 23:00:22.447587 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.447568 2564 scope.go:117] "RemoveContainer" containerID="6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af" Jul 06 23:00:22.447852 ip-10-0-135-55 kubenswrapper[2564]: E0706 23:00:22.447832 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af\": container with ID starting with 6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af not found: ID does not exist" containerID="6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af" Jul 06 23:00:22.447925 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.447867 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af"} err="failed to get container status \"6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af\": rpc error: code = NotFound desc = could not find container \"6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af\": container with ID starting with 6536218bc89e91b39032d14a91dbcaf239cee39ab92ef0b1c0d42e39360b03af not found: ID does not exist" Jul 06 23:00:22.537285 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537246 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-home\") pod \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " Jul 06 23:00:22.537285 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-dshm\") pod \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " Jul 06 23:00:22.537569 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tls-certs\") pod \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " Jul 06 23:00:22.537569 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537347 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-model-cache\") pod \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " Jul 06 23:00:22.537569 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537372 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tmp-dir\") pod \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " Jul 06 23:00:22.537569 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537402 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9g7m\" (UniqueName: \"kubernetes.io/projected/e539e2e4-5a4f-4282-8628-bf10e656c7c7-kube-api-access-j9g7m\") pod \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\" (UID: \"e539e2e4-5a4f-4282-8628-bf10e656c7c7\") " Jul 06 23:00:22.537758 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537568 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-home" (OuterVolumeSpecName: "home") pod "e539e2e4-5a4f-4282-8628-bf10e656c7c7" (UID: "e539e2e4-5a4f-4282-8628-bf10e656c7c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:22.537758 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537679 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e539e2e4-5a4f-4282-8628-bf10e656c7c7" (UID: "e539e2e4-5a4f-4282-8628-bf10e656c7c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:22.537758 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537693 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e539e2e4-5a4f-4282-8628-bf10e656c7c7" (UID: "e539e2e4-5a4f-4282-8628-bf10e656c7c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:22.537866 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537854 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:22.537901 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537873 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:22.537947 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.537912 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:22.539676 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.539648 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e539e2e4-5a4f-4282-8628-bf10e656c7c7" (UID: "e539e2e4-5a4f-4282-8628-bf10e656c7c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:22.539801 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.539719 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-dshm" (OuterVolumeSpecName: "dshm") pod "e539e2e4-5a4f-4282-8628-bf10e656c7c7" (UID: "e539e2e4-5a4f-4282-8628-bf10e656c7c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:22.539801 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.539743 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e539e2e4-5a4f-4282-8628-bf10e656c7c7-kube-api-access-j9g7m" (OuterVolumeSpecName: "kube-api-access-j9g7m") pod "e539e2e4-5a4f-4282-8628-bf10e656c7c7" (UID: "e539e2e4-5a4f-4282-8628-bf10e656c7c7"). InnerVolumeSpecName "kube-api-access-j9g7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:22.638416 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.638338 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9g7m\" (UniqueName: \"kubernetes.io/projected/e539e2e4-5a4f-4282-8628-bf10e656c7c7-kube-api-access-j9g7m\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:22.638416 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.638364 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e539e2e4-5a4f-4282-8628-bf10e656c7c7-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:22.638416 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.638378 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e539e2e4-5a4f-4282-8628-bf10e656c7c7-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:22.759652 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.759622 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9"] Jul 06 23:00:22.766122 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.766095 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/auth-test-model-936b5d-kserve-6f8d57f487-4q4r9"] Jul 06 23:00:22.810247 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.810206 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e539e2e4-5a4f-4282-8628-bf10e656c7c7" path="/var/lib/kubelet/pods/e539e2e4-5a4f-4282-8628-bf10e656c7c7/volumes" Jul 06 23:00:22.939573 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:22.939480 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" podUID="869ba53f-531e-4be9-9513-724732594a70" containerName="main" containerID="cri-o://bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f" gracePeriod=45 Jul 06 23:00:23.213738 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.213704 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 23:00:23.344311 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344282 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-dshm\") pod \"869ba53f-531e-4be9-9513-724732594a70\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " Jul 06 23:00:23.344533 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-tmp-dir\") pod \"869ba53f-531e-4be9-9513-724732594a70\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " Jul 06 23:00:23.344533 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344350 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-model-cache\") pod \"869ba53f-531e-4be9-9513-724732594a70\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " Jul 06 23:00:23.344533 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344376 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-home\") pod \"869ba53f-531e-4be9-9513-724732594a70\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " Jul 06 23:00:23.344533 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344399 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6lq6\" (UniqueName: \"kubernetes.io/projected/869ba53f-531e-4be9-9513-724732594a70-kube-api-access-c6lq6\") pod \"869ba53f-531e-4be9-9513-724732594a70\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " Jul 06 23:00:23.344533 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344483 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869ba53f-531e-4be9-9513-724732594a70-tls-certs\") pod \"869ba53f-531e-4be9-9513-724732594a70\" (UID: \"869ba53f-531e-4be9-9513-724732594a70\") " Jul 06 23:00:23.344812 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344631 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "869ba53f-531e-4be9-9513-724732594a70" (UID: "869ba53f-531e-4be9-9513-724732594a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:23.344812 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344645 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-model-cache" (OuterVolumeSpecName: "model-cache") pod "869ba53f-531e-4be9-9513-724732594a70" (UID: "869ba53f-531e-4be9-9513-724732594a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:23.344812 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344659 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-home" (OuterVolumeSpecName: "home") pod "869ba53f-531e-4be9-9513-724732594a70" (UID: "869ba53f-531e-4be9-9513-724732594a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:23.344937 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344844 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:23.344937 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344862 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:23.344937 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.344876 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:23.346886 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.346858 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-dshm" (OuterVolumeSpecName: "dshm") pod "869ba53f-531e-4be9-9513-724732594a70" (UID: "869ba53f-531e-4be9-9513-724732594a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:00:23.347017 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.346882 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869ba53f-531e-4be9-9513-724732594a70-kube-api-access-c6lq6" (OuterVolumeSpecName: "kube-api-access-c6lq6") pod "869ba53f-531e-4be9-9513-724732594a70" (UID: "869ba53f-531e-4be9-9513-724732594a70"). InnerVolumeSpecName "kube-api-access-c6lq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:00:23.347017 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.346967 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869ba53f-531e-4be9-9513-724732594a70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "869ba53f-531e-4be9-9513-724732594a70" (UID: "869ba53f-531e-4be9-9513-724732594a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:00:23.445186 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445093 2564 generic.go:356] "Generic (PLEG): container finished" podID="869ba53f-531e-4be9-9513-724732594a70" containerID="bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f" exitCode=0 Jul 06 23:00:23.445186 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445141 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" Jul 06 23:00:23.445406 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" event={"ID":"869ba53f-531e-4be9-9513-724732594a70","Type":"ContainerDied","Data":"bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f"} Jul 06 23:00:23.445406 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6" event={"ID":"869ba53f-531e-4be9-9513-724732594a70","Type":"ContainerDied","Data":"47d63b430e4621042bec43803a413d8cce826fbfcfa80075677252b603016896"} Jul 06 23:00:23.445406 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445234 2564 scope.go:117] "RemoveContainer" containerID="bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f" Jul 06 23:00:23.445564 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445411 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/869ba53f-531e-4be9-9513-724732594a70-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:23.445564 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445473 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6lq6\" (UniqueName: \"kubernetes.io/projected/869ba53f-531e-4be9-9513-724732594a70-kube-api-access-c6lq6\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:23.445564 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.445484 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/869ba53f-531e-4be9-9513-724732594a70-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:00:23.454486 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.454435 2564 scope.go:117] "RemoveContainer" containerID="bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f" Jul 06 23:00:23.454725 ip-10-0-135-55 kubenswrapper[2564]: E0706 23:00:23.454700 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f\": container with ID starting with bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f not found: ID does not exist" containerID="bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f" Jul 06 23:00:23.454821 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.454730 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f"} err="failed to get container status \"bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f\": rpc error: code = NotFound desc = could not find container \"bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f\": container with ID starting with bb7dc0620043ea6f03a85cbf87823e0c81b04e790b7d44f728150e9e2011919f not found: ID does not exist" Jul 06 23:00:23.465850 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.465802 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6"] Jul 06 23:00:23.467592 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:23.467571 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/auth-test-model-cf2b54-kserve-84dd469d8c-gzqp6"] Jul 06 23:00:24.810414 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:24.810381 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="869ba53f-531e-4be9-9513-724732594a70" path="/var/lib/kubelet/pods/869ba53f-531e-4be9-9513-724732594a70/volumes" Jul 06 23:00:54.315737 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.315703 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff"] Jul 06 23:00:54.316326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.316076 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="869ba53f-531e-4be9-9513-724732594a70" containerName="main" Jul 06 23:00:54.316326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.316104 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="869ba53f-531e-4be9-9513-724732594a70" containerName="main" Jul 06 23:00:54.316326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.316118 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e539e2e4-5a4f-4282-8628-bf10e656c7c7" containerName="main" Jul 06 23:00:54.316326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.316124 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e539e2e4-5a4f-4282-8628-bf10e656c7c7" containerName="main" Jul 06 23:00:54.316326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.316200 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="869ba53f-531e-4be9-9513-724732594a70" containerName="main" Jul 06 23:00:54.316326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.316210 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e539e2e4-5a4f-4282-8628-bf10e656c7c7" containerName="main" Jul 06 23:00:54.319007 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.318990 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.321833 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.321806 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"openshift-service-ca.crt\"" Jul 06 23:00:54.323015 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.322995 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"default-dockercfg-s796x\"" Jul 06 23:00:54.323196 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.323167 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"kube-root-ca.crt\"" Jul 06 23:00:54.323196 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.323191 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-cf2b54\"/\"sub-test-model-cf2b54-kserve-self-signed-certs\"" Jul 06 23:00:54.331053 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.331023 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff"] Jul 06 23:00:54.400021 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.399990 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:00:54.421254 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.421216 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-tmp-dir\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.421502 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.421277 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5k67\" (UniqueName: \"kubernetes.io/projected/a44418b3-6276-4146-8afb-c376f146d703-kube-api-access-m5k67\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.421502 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.421360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-dshm\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.421502 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.421387 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-home\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.421502 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.421410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-model-cache\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.421670 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.421515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44418b3-6276-4146-8afb-c376f146d703-tls-certs\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.521974 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.521940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-dshm\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.521974 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.521982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-home\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522252 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522001 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-model-cache\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522252 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44418b3-6276-4146-8afb-c376f146d703-tls-certs\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522252 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-tmp-dir\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522252 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5k67\" (UniqueName: \"kubernetes.io/projected/a44418b3-6276-4146-8afb-c376f146d703-kube-api-access-m5k67\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522558 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522466 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-home\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522558 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522516 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-model-cache\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.522558 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.522528 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-tmp-dir\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.524413 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.524391 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-dshm\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.524562 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.524547 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44418b3-6276-4146-8afb-c376f146d703-tls-certs\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.530654 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.530627 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5k67\" (UniqueName: \"kubernetes.io/projected/a44418b3-6276-4146-8afb-c376f146d703-kube-api-access-m5k67\") pod \"sub-test-model-cf2b54-kserve-5696d7f447-c6vff\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.629536 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.629500 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:00:54.773355 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:54.773327 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff"] Jul 06 23:00:54.775502 ip-10-0-135-55 kubenswrapper[2564]: W0706 23:00:54.775470 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda44418b3_6276_4146_8afb_c376f146d703.slice/crio-57e248dde5aa70e6f91719a83345bc159b1deec8919f4d1867bdf75abb1c5a98 WatchSource:0}: Error finding container 57e248dde5aa70e6f91719a83345bc159b1deec8919f4d1867bdf75abb1c5a98: Status 404 returned error can't find the container with id 57e248dde5aa70e6f91719a83345bc159b1deec8919f4d1867bdf75abb1c5a98 Jul 06 23:00:55.602481 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:55.602445 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" event={"ID":"a44418b3-6276-4146-8afb-c376f146d703","Type":"ContainerStarted","Data":"20b9e1a6a4d79ab09f5a1d3986c22ea86a0364c06bac7709701e0df208f70e92"} Jul 06 23:00:55.602481 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:55.602486 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" event={"ID":"a44418b3-6276-4146-8afb-c376f146d703","Type":"ContainerStarted","Data":"57e248dde5aa70e6f91719a83345bc159b1deec8919f4d1867bdf75abb1c5a98"} Jul 06 23:00:55.619349 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:00:55.619298 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" podStartSLOduration=1.619283042 podStartE2EDuration="1.619283042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:00:55.617359129 +0000 UTC m=+2217.353761116" watchObservedRunningTime="2026-07-06 23:00:55.619283042 +0000 UTC m=+2217.355685028" Jul 06 23:01:04.629966 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:04.629934 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:01:04.630473 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:04.629982 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:01:04.643164 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:04.643139 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:01:04.656940 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:04.656915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:01:45.022820 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:45.022775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:45.783837 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:45.783807 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff"] Jul 06 23:01:45.802642 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:45.802608 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:52.682572 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:52.682515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:54.507686 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.507649 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5"] Jul 06 23:01:54.511898 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.511873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.515608 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.515584 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"kube-root-ca.crt\"" Jul 06 23:01:54.515845 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.515604 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"default-dockercfg-cl27v\"" Jul 06 23:01:54.515924 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.515608 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"openshift-service-ca.crt\"" Jul 06 23:01:54.515988 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.515626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-936b5d\"/\"rate-test-model-936b5d-kserve-self-signed-certs\"" Jul 06 23:01:54.519995 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.519414 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5"] Jul 06 23:01:54.696570 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.696535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-dshm\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.696727 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.696618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tls-certs\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.696727 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.696641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4jk\" (UniqueName: \"kubernetes.io/projected/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-kube-api-access-6d4jk\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.696727 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.696671 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-home\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.696874 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.696733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-model-cache\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.696874 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.696767 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tmp-dir\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.797746 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.797657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tmp-dir\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.797746 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.797717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-dshm\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.797949 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.797782 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tls-certs\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.797949 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.797809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4jk\" (UniqueName: \"kubernetes.io/projected/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-kube-api-access-6d4jk\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.797949 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.797845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-home\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.797949 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.797880 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-model-cache\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.798166 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.798140 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tmp-dir\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.798263 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.798240 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-model-cache\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.798382 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.798310 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-home\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.800253 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.800219 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-dshm\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.801260 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.800638 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tls-certs\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.806622 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.806599 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4jk\" (UniqueName: \"kubernetes.io/projected/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-kube-api-access-6d4jk\") pod \"rate-test-model-936b5d-kserve-787765c85f-fv8k5\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.823027 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.822997 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:01:54.965501 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:54.965468 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5"] Jul 06 23:01:54.967440 ip-10-0-135-55 kubenswrapper[2564]: W0706 23:01:54.967406 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07bd8d1d_1e1f_4a9c_be42_630285cc90eb.slice/crio-bf9cd4013e9b9d4d14be0ac4dca3d9ac5bfba60255fd9655a3a04e5990867d72 WatchSource:0}: Error finding container bf9cd4013e9b9d4d14be0ac4dca3d9ac5bfba60255fd9655a3a04e5990867d72: Status 404 returned error can't find the container with id bf9cd4013e9b9d4d14be0ac4dca3d9ac5bfba60255fd9655a3a04e5990867d72 Jul 06 23:01:55.107545 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:55.107510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:01:55.901793 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:55.901749 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" event={"ID":"07bd8d1d-1e1f-4a9c-be42-630285cc90eb","Type":"ContainerStarted","Data":"b9b7ef9f036a98184a8009b701b9927666e4a59d5fdbd87d615326479a19df2b"} Jul 06 23:01:55.902208 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:55.901806 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" event={"ID":"07bd8d1d-1e1f-4a9c-be42-630285cc90eb","Type":"ContainerStarted","Data":"bf9cd4013e9b9d4d14be0ac4dca3d9ac5bfba60255fd9655a3a04e5990867d72"} Jul 06 23:01:55.917138 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:01:55.917086 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" podStartSLOduration=1.9170665439999999 podStartE2EDuration="1.917066544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 23:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:01:55.916529407 +0000 UTC m=+2277.652931393" watchObservedRunningTime="2026-07-06 23:01:55.917066544 +0000 UTC m=+2277.653468531" Jul 06 23:02:00.806483 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:00.806407 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" podUID="a44418b3-6276-4146-8afb-c376f146d703" containerName="main" containerID="cri-o://20b9e1a6a4d79ab09f5a1d3986c22ea86a0364c06bac7709701e0df208f70e92" gracePeriod=45 Jul 06 23:02:00.927073 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:00.927042 2564 generic.go:356] "Generic (PLEG): container finished" podID="a44418b3-6276-4146-8afb-c376f146d703" containerID="20b9e1a6a4d79ab09f5a1d3986c22ea86a0364c06bac7709701e0df208f70e92" exitCode=0 Jul 06 23:02:00.927247 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:00.927108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" event={"ID":"a44418b3-6276-4146-8afb-c376f146d703","Type":"ContainerDied","Data":"20b9e1a6a4d79ab09f5a1d3986c22ea86a0364c06bac7709701e0df208f70e92"} Jul 06 23:02:01.067349 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.067276 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:02:01.157203 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157169 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5k67\" (UniqueName: \"kubernetes.io/projected/a44418b3-6276-4146-8afb-c376f146d703-kube-api-access-m5k67\") pod \"a44418b3-6276-4146-8afb-c376f146d703\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " Jul 06 23:02:01.157203 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157208 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-model-cache\") pod \"a44418b3-6276-4146-8afb-c376f146d703\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " Jul 06 23:02:01.157421 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44418b3-6276-4146-8afb-c376f146d703-tls-certs\") pod \"a44418b3-6276-4146-8afb-c376f146d703\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " Jul 06 23:02:01.157421 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157360 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-tmp-dir\") pod \"a44418b3-6276-4146-8afb-c376f146d703\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " Jul 06 23:02:01.157421 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157401 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-dshm\") pod \"a44418b3-6276-4146-8afb-c376f146d703\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " Jul 06 23:02:01.157583 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157461 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-home\") pod \"a44418b3-6276-4146-8afb-c376f146d703\" (UID: \"a44418b3-6276-4146-8afb-c376f146d703\") " Jul 06 23:02:01.157583 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157543 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-model-cache" (OuterVolumeSpecName: "model-cache") pod "a44418b3-6276-4146-8afb-c376f146d703" (UID: "a44418b3-6276-4146-8afb-c376f146d703"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:01.157697 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157623 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a44418b3-6276-4146-8afb-c376f146d703" (UID: "a44418b3-6276-4146-8afb-c376f146d703"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:01.157758 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157730 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-home" (OuterVolumeSpecName: "home") pod "a44418b3-6276-4146-8afb-c376f146d703" (UID: "a44418b3-6276-4146-8afb-c376f146d703"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:01.157855 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157836 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:02:01.157855 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157853 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:02:01.158038 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.157862 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:02:01.159451 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.159401 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44418b3-6276-4146-8afb-c376f146d703-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a44418b3-6276-4146-8afb-c376f146d703" (UID: "a44418b3-6276-4146-8afb-c376f146d703"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:02:01.159451 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.159415 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-dshm" (OuterVolumeSpecName: "dshm") pod "a44418b3-6276-4146-8afb-c376f146d703" (UID: "a44418b3-6276-4146-8afb-c376f146d703"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:02:01.159587 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.159530 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44418b3-6276-4146-8afb-c376f146d703-kube-api-access-m5k67" (OuterVolumeSpecName: "kube-api-access-m5k67") pod "a44418b3-6276-4146-8afb-c376f146d703" (UID: "a44418b3-6276-4146-8afb-c376f146d703"). InnerVolumeSpecName "kube-api-access-m5k67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:02:01.258727 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.258688 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a44418b3-6276-4146-8afb-c376f146d703-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:02:01.258727 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.258721 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a44418b3-6276-4146-8afb-c376f146d703-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:02:01.258727 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.258731 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5k67\" (UniqueName: \"kubernetes.io/projected/a44418b3-6276-4146-8afb-c376f146d703-kube-api-access-m5k67\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:02:01.933259 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.933231 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" Jul 06 23:02:01.933259 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.933245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff" event={"ID":"a44418b3-6276-4146-8afb-c376f146d703","Type":"ContainerDied","Data":"57e248dde5aa70e6f91719a83345bc159b1deec8919f4d1867bdf75abb1c5a98"} Jul 06 23:02:01.933808 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.933289 2564 scope.go:117] "RemoveContainer" containerID="20b9e1a6a4d79ab09f5a1d3986c22ea86a0364c06bac7709701e0df208f70e92" Jul 06 23:02:01.962973 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.962942 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff"] Jul 06 23:02:01.970063 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:01.970035 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-cf2b54/sub-test-model-cf2b54-kserve-5696d7f447-c6vff"] Jul 06 23:02:02.810740 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:02.810705 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44418b3-6276-4146-8afb-c376f146d703" path="/var/lib/kubelet/pods/a44418b3-6276-4146-8afb-c376f146d703/volumes" Jul 06 23:02:04.823921 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:04.823882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:02:04.824336 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:04.823968 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:02:04.836860 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:04.836826 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:02:04.959873 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:04.959846 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:02:07.408275 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:07.408249 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:07.508660 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:07.508626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:23.391767 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:23.391733 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:23.491870 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:23.491829 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:25.152939 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:25.152901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:30.699149 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:30.699107 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:30.797835 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:30.797795 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:02:33.332733 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:02:33.332702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:20.133335 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:20.133300 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5"] Jul 06 23:03:20.154092 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:20.154063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:21.627392 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:21.627348 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:22.038726 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:22.038636 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:22.484260 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:22.484226 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:22.588706 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:22.588674 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:22.688677 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:22.688642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:22.820847 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:22.820763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:27.519616 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:27.519580 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:03:35.153668 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.153628 2564 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" podUID="07bd8d1d-1e1f-4a9c-be42-630285cc90eb" containerName="main" containerID="cri-o://b9b7ef9f036a98184a8009b701b9927666e4a59d5fdbd87d615326479a19df2b" gracePeriod=45 Jul 06 23:03:35.393988 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.393962 2564 generic.go:356] "Generic (PLEG): container finished" podID="07bd8d1d-1e1f-4a9c-be42-630285cc90eb" containerID="b9b7ef9f036a98184a8009b701b9927666e4a59d5fdbd87d615326479a19df2b" exitCode=0 Jul 06 23:03:35.394101 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.394031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" event={"ID":"07bd8d1d-1e1f-4a9c-be42-630285cc90eb","Type":"ContainerDied","Data":"b9b7ef9f036a98184a8009b701b9927666e4a59d5fdbd87d615326479a19df2b"} Jul 06 23:03:35.413937 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.413867 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:03:35.512083 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512048 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-model-cache\") pod \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " Jul 06 23:03:35.512083 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512091 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6d4jk\" (UniqueName: \"kubernetes.io/projected/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-kube-api-access-6d4jk\") pod \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " Jul 06 23:03:35.512359 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512115 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tmp-dir\") pod \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " Jul 06 23:03:35.512359 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512148 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tls-certs\") pod \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " Jul 06 23:03:35.512359 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512193 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-dshm\") pod \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " Jul 06 23:03:35.512359 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512208 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-home\") pod \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\" (UID: \"07bd8d1d-1e1f-4a9c-be42-630285cc90eb\") " Jul 06 23:03:35.512594 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512407 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "07bd8d1d-1e1f-4a9c-be42-630285cc90eb" (UID: "07bd8d1d-1e1f-4a9c-be42-630285cc90eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:35.512594 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512480 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07bd8d1d-1e1f-4a9c-be42-630285cc90eb" (UID: "07bd8d1d-1e1f-4a9c-be42-630285cc90eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:35.512594 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.512559 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-home" (OuterVolumeSpecName: "home") pod "07bd8d1d-1e1f-4a9c-be42-630285cc90eb" (UID: "07bd8d1d-1e1f-4a9c-be42-630285cc90eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:35.514406 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.514373 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-dshm" (OuterVolumeSpecName: "dshm") pod "07bd8d1d-1e1f-4a9c-be42-630285cc90eb" (UID: "07bd8d1d-1e1f-4a9c-be42-630285cc90eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 23:03:35.514568 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.514420 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-kube-api-access-6d4jk" (OuterVolumeSpecName: "kube-api-access-6d4jk") pod "07bd8d1d-1e1f-4a9c-be42-630285cc90eb" (UID: "07bd8d1d-1e1f-4a9c-be42-630285cc90eb"). InnerVolumeSpecName "kube-api-access-6d4jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 23:03:35.514568 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.514458 2564 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07bd8d1d-1e1f-4a9c-be42-630285cc90eb" (UID: "07bd8d1d-1e1f-4a9c-be42-630285cc90eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 23:03:35.613866 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.613829 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-model-cache\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:03:35.613866 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.613858 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4jk\" (UniqueName: \"kubernetes.io/projected/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-kube-api-access-6d4jk\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:03:35.614082 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.613890 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tmp-dir\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:03:35.614082 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.613900 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-tls-certs\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:03:35.614082 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.613909 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-dshm\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:03:35.614082 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:35.613917 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07bd8d1d-1e1f-4a9c-be42-630285cc90eb-home\") on node \"ip-10-0-135-55.ec2.internal\" DevicePath \"\"" Jul 06 23:03:36.400493 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:36.400459 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" Jul 06 23:03:36.400997 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:36.400493 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5" event={"ID":"07bd8d1d-1e1f-4a9c-be42-630285cc90eb","Type":"ContainerDied","Data":"bf9cd4013e9b9d4d14be0ac4dca3d9ac5bfba60255fd9655a3a04e5990867d72"} Jul 06 23:03:36.400997 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:36.400556 2564 scope.go:117] "RemoveContainer" containerID="b9b7ef9f036a98184a8009b701b9927666e4a59d5fdbd87d615326479a19df2b" Jul 06 23:03:36.421906 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:36.421870 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5"] Jul 06 23:03:36.427402 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:36.427376 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-936b5d/rate-test-model-936b5d-kserve-787765c85f-fv8k5"] Jul 06 23:03:36.810622 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:36.810544 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07bd8d1d-1e1f-4a9c-be42-630285cc90eb" path="/var/lib/kubelet/pods/07bd8d1d-1e1f-4a9c-be42-630285cc90eb/volumes" Jul 06 23:03:58.869843 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:58.869806 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 23:03:58.877091 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:03:58.877059 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-4fhw9_0827b7d5-e529-4e47-9882-b207e32dc594/ovn-acl-logging/0.log" Jul 06 23:04:04.562477 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:04.562447 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:04.835065 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:04.835032 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:05.051434 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:05.051392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:05.198890 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:05.198799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:05.612234 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:05.612203 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:05.708712 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:05.708681 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:05.854371 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:05.854337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:11.605774 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:11.605741 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:11.667070 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:11.667034 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:11.904116 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:11.904027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:12.113233 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:12.113194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:12.522881 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:12.522841 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f6989dd7f-46trd_a89c4d84-97d0-4b6d-b755-09c131839a2e/authorino/0.log" Jul 06 23:04:16.980796 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:16.980765 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xthph_4e0a57f9-b003-45f9-86a9-b0752d1aede5/manager/0.log" Jul 06 23:04:17.679918 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:17.679891 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6c44h_17b9b2b7-b744-4ed2-a6ca-cd4976cec84c/server/0.log" Jul 06 23:04:17.795147 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:17.795117 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2dkjj_5b717620-cdd9-4e9d-bc89-7280bf8108fe/manager/0.log" Jul 06 23:04:18.078962 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:18.078935 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:18.138885 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:18.138850 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:18.168479 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:18.168451 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84b8787b6c-kc67b_cbd4a953-28be-49c0-a00f-9508dc0b595e/manager/0.log" Jul 06 23:04:18.178682 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:18.178660 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84b8787b6c-kc67b_cbd4a953-28be-49c0-a00f-9508dc0b595e/copy-manifests/0.log" Jul 06 23:04:19.592331 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:19.592302 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f6989dd7f-46trd_a89c4d84-97d0-4b6d-b755-09c131839a2e/authorino/0.log" Jul 06 23:04:19.716539 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:19.716512 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w8f97_af9bacca-4e0f-4dab-8bf9-f55af61cee27/manager/0.log" Jul 06 23:04:19.825892 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:19.825866 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7vxgd_49b18a4b-4dd0-464d-b8df-49a3448248fc/manager/0.log" Jul 06 23:04:20.060357 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:20.060281 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zwcc4_d4fa971d-7b06-4efc-9427-a69e25594da7/registry-server/0.log" Jul 06 23:04:20.177274 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:20.177240 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bmnx9_239e30b9-59f5-4e4f-808b-1c33804b5dc4/manager/0.log" Jul 06 23:04:20.406784 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:20.406750 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rm27b_39f966d8-5e63-484d-a7f1-1621a3ad0b69/manager/0.log" Jul 06 23:04:20.747488 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:20.747396 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vkz5j_b830ff70-f9c1-45be-b0b8-b2fcc42bb772/istio-proxy/0.log" Jul 06 23:04:20.976056 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:20.976026 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6b7b4fbb-6t9nm_a3157ed2-0dff-4199-8764-478bc8e591db/kube-auth-proxy/0.log" Jul 06 23:04:21.211403 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:21.211371 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/1.log" Jul 06 23:04:21.554092 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:21.554004 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd494bcf9-kfm22_24907fdd-7f13-4aeb-bdde-365f931c8c82/router/0.log" Jul 06 23:04:21.886345 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:21.886316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-4s4m2_4a462b2d-2944-41c7-9531-38cb4dc59d6d/main/0.log" Jul 06 23:04:22.331108 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:22.331080 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-crfsr_7c875ba1-564a-4e2a-bd4a-e7bb1fa93ff2/main/0.log" Jul 06 23:04:22.443045 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:22.443017 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-fz7p6_ffbe7298-4203-4be7-9184-9eb5bb4e27d8/main/0.log" Jul 06 23:04:24.862495 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:24.862459 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:26.723799 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:26.723767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:26.793439 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:26.793392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:29.472898 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:29.472868 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-qb6jc_9a70d449-b56c-4f5b-8af3-90734fa84198/global-pull-secret-syncer/0.log" Jul 06 23:04:29.570444 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:29.570403 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" Jul 06 23:04:29.746012 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:29.745934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-55.ec2.internal_86df2e047708b9ac3980718e830e2fcf/haproxy/0.log" Jul 06 23:04:34.750039 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:34.749934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f6989dd7f-46trd_a89c4d84-97d0-4b6d-b755-09c131839a2e/authorino/0.log" Jul 06 23:04:34.830277 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:34.830248 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-w8f97_af9bacca-4e0f-4dab-8bf9-f55af61cee27/manager/0.log" Jul 06 23:04:34.890210 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:34.890184 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-7vxgd_49b18a4b-4dd0-464d-b8df-49a3448248fc/manager/0.log" Jul 06 23:04:34.972785 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:34.972750 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zwcc4_d4fa971d-7b06-4efc-9427-a69e25594da7/registry-server/0.log" Jul 06 23:04:35.061987 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:35.061894 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-bmnx9_239e30b9-59f5-4e4f-808b-1c33804b5dc4/manager/0.log" Jul 06 23:04:35.152029 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:35.151976 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rm27b_39f966d8-5e63-484d-a7f1-1621a3ad0b69/manager/0.log" Jul 06 23:04:37.593439 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:37.593396 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt4nd_fbe9c034-1f93-4690-9202-4e72ff368100/node-exporter/0.log" Jul 06 23:04:37.618611 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:37.618585 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt4nd_fbe9c034-1f93-4690-9202-4e72ff368100/kube-rbac-proxy/0.log" Jul 06 23:04:37.649280 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:37.649250 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wt4nd_fbe9c034-1f93-4690-9202-4e72ff368100/init-textfile/0.log" Jul 06 23:04:40.222073 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.222030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:40.241215 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241174 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn"] Jul 06 23:04:40.241604 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241579 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07bd8d1d-1e1f-4a9c-be42-630285cc90eb" containerName="main" Jul 06 23:04:40.241604 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241607 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="07bd8d1d-1e1f-4a9c-be42-630285cc90eb" containerName="main" Jul 06 23:04:40.241707 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241616 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44418b3-6276-4146-8afb-c376f146d703" containerName="main" Jul 06 23:04:40.241707 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241622 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44418b3-6276-4146-8afb-c376f146d703" containerName="main" Jul 06 23:04:40.241707 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241682 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44418b3-6276-4146-8afb-c376f146d703" containerName="main" Jul 06 23:04:40.241707 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.241695 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="07bd8d1d-1e1f-4a9c-be42-630285cc90eb" containerName="main" Jul 06 23:04:40.244669 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.244650 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.247746 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.247717 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-768zc\"/\"kube-root-ca.crt\"" Jul 06 23:04:40.247746 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.247732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-768zc\"/\"default-dockercfg-ks99f\"" Jul 06 23:04:40.247746 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.247717 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-768zc\"/\"openshift-service-ca.crt\"" Jul 06 23:04:40.253960 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.253934 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn"] Jul 06 23:04:40.298162 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.298116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-podres\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.298162 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.298167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-proc\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.298404 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.298248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-lib-modules\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.298404 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.298294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjjhp\" (UniqueName: \"kubernetes.io/projected/cec61344-b163-4795-808b-de888bf5670e-kube-api-access-fjjhp\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.298404 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.298323 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-sys\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.323321 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.323289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6kkfd_94be3bea-576a-445b-ae39-58e3dff104de/konnectivity-agent/0.log" limit=1024 Jul 06 23:04:40.398822 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.398781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-lib-modules\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399024 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.398842 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjjhp\" (UniqueName: \"kubernetes.io/projected/cec61344-b163-4795-808b-de888bf5670e-kube-api-access-fjjhp\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399024 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.398875 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-sys\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399024 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.398960 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-podres\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399024 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.398971 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-lib-modules\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399178 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.399044 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-sys\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399178 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.399066 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-proc\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399178 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.399049 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-podres\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.399178 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.399108 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cec61344-b163-4795-808b-de888bf5670e-proc\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.407179 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.407135 2564 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjjhp\" (UniqueName: \"kubernetes.io/projected/cec61344-b163-4795-808b-de888bf5670e-kube-api-access-fjjhp\") pod \"perf-node-gather-daemonset-8t4xn\" (UID: \"cec61344-b163-4795-808b-de888bf5670e\") " pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.556108 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.556005 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:40.695305 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:40.695271 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn"] Jul 06 23:04:40.697978 ip-10-0-135-55 kubenswrapper[2564]: W0706 23:04:40.697942 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcec61344_b163_4795_808b_de888bf5670e.slice/crio-9ef29b974facb8ac591277839f24e431f272dde3d40d41f9d1c1ec73d5bf3600 WatchSource:0}: Error finding container 9ef29b974facb8ac591277839f24e431f272dde3d40d41f9d1c1ec73d5bf3600: Status 404 returned error can't find the container with id 9ef29b974facb8ac591277839f24e431f272dde3d40d41f9d1c1ec73d5bf3600 Jul 06 23:04:41.367538 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:41.367507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-wzcsx_752a5490-71cf-4d8b-830a-2eb5c9560e5f/volume-data-source-validator/0.log" Jul 06 23:04:41.712354 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:41.712242 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" event={"ID":"cec61344-b163-4795-808b-de888bf5670e","Type":"ContainerStarted","Data":"7028a0dd05dab06fca5a805eb864ce2f1fe3c8d595c16585a395cc681e9beb17"} Jul 06 23:04:41.712354 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:41.712306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" event={"ID":"cec61344-b163-4795-808b-de888bf5670e","Type":"ContainerStarted","Data":"9ef29b974facb8ac591277839f24e431f272dde3d40d41f9d1c1ec73d5bf3600"} Jul 06 23:04:41.712624 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:41.712401 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:41.729550 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:41.729483 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" podStartSLOduration=1.7294599339999999 podStartE2EDuration="1.729459934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 23:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 23:04:41.727864111 +0000 UTC m=+2443.464266098" watchObservedRunningTime="2026-07-06 23:04:41.729459934 +0000 UTC m=+2443.465861937" Jul 06 23:04:42.474617 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:42.474577 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ftjg_1689f60b-7849-452d-ba1a-27a8057595a9/dns/0.log" Jul 06 23:04:42.496326 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:42.496304 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ftjg_1689f60b-7849-452d-ba1a-27a8057595a9/kube-rbac-proxy/0.log" Jul 06 23:04:42.613929 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:42.613901 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nsz97_caaf6673-9f01-40c6-a0ee-fe4992b9385a/dns-node-resolver/0.log" Jul 06 23:04:43.171342 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:43.171313 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7774bcc-qfphc_f330c5c0-7c8e-47b4-8924-7414a02f502e/registry/0.log" Jul 06 23:04:43.217197 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:43.217165 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g8jwm_b0aa773f-d298-4bde-a4fb-95518c2339ac/node-ca/0.log" Jul 06 23:04:44.228559 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:44.228534 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vkz5j_b830ff70-f9c1-45be-b0b8-b2fcc42bb772/istio-proxy/0.log" Jul 06 23:04:44.472926 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:44.472898 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6b7b4fbb-6t9nm_a3157ed2-0dff-4199-8764-478bc8e591db/kube-auth-proxy/0.log" Jul 06 23:04:44.580719 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:44.580695 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/0.log" Jul 06 23:04:44.606837 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:44.606803 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/1.log" Jul 06 23:04:44.747902 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:44.747869 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd494bcf9-kfm22_24907fdd-7f13-4aeb-bdde-365f931c8c82/router/0.log" Jul 06 23:04:45.492248 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:45.492220 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lfhr9_f21331f3-0785-4537-bcd4-9151b6de0dea/serve-healthcheck-canary/0.log" Jul 06 23:04:46.038086 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:46.038059 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bhpss_8382f3d5-bbd9-4ce9-9b35-81a3debfa308/kube-rbac-proxy/0.log" Jul 06 23:04:46.059708 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:46.059683 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bhpss_8382f3d5-bbd9-4ce9-9b35-81a3debfa308/exporter/0.log" Jul 06 23:04:46.081220 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:46.081195 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bhpss_8382f3d5-bbd9-4ce9-9b35-81a3debfa308/extractor/0.log" Jul 06 23:04:47.727271 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:47.727243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-768zc/perf-node-gather-daemonset-8t4xn" Jul 06 23:04:48.519534 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:48.519507 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xthph_4e0a57f9-b003-45f9-86a9-b0752d1aede5/manager/0.log" Jul 06 23:04:48.963460 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:48.963415 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6c44h_17b9b2b7-b744-4ed2-a6ca-cd4976cec84c/server/0.log" Jul 06 23:04:49.028092 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:49.028060 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-2dkjj_5b717620-cdd9-4e9d-bc89-7280bf8108fe/manager/0.log" Jul 06 23:04:49.251050 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:49.250962 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84b8787b6c-kc67b_cbd4a953-28be-49c0-a00f-9508dc0b595e/manager/0.log" Jul 06 23:04:49.276627 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:49.276598 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-84b8787b6c-kc67b_cbd4a953-28be-49c0-a00f-9508dc0b595e/copy-manifests/0.log" Jul 06 23:04:50.928437 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:50.928397 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vkz5j_b830ff70-f9c1-45be-b0b8-b2fcc42bb772/istio-proxy/0.log" Jul 06 23:04:51.119396 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:51.119357 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6b7b4fbb-6t9nm_a3157ed2-0dff-4199-8764-478bc8e591db/kube-auth-proxy/0.log" Jul 06 23:04:51.158216 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:51.158182 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/0.log" Jul 06 23:04:51.190862 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:51.190766 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/1.log" Jul 06 23:04:51.280380 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:51.280350 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd494bcf9-kfm22_24907fdd-7f13-4aeb-bdde-365f931c8c82/router/0.log" Jul 06 23:04:52.551828 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:52.551791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4vkz5j_b830ff70-f9c1-45be-b0b8-b2fcc42bb772/istio-proxy/0.log" Jul 06 23:04:52.741078 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:52.741049 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c6b7b4fbb-6t9nm_a3157ed2-0dff-4199-8764-478bc8e591db/kube-auth-proxy/0.log" Jul 06 23:04:52.789512 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:52.789480 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/0.log" Jul 06 23:04:52.819323 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:52.819241 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g8qkz_e3a0dda1-4e7e-45b4-9b66-4eef140d371b/istio-proxy/1.log" Jul 06 23:04:52.908319 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:52.908288 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5fd494bcf9-kfm22_24907fdd-7f13-4aeb-bdde-365f931c8c82/router/0.log" Jul 06 23:04:57.377233 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.377202 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5j788_2172046f-3f2b-40c3-be14-dac49e1160bd/kube-multus/0.log" Jul 06 23:04:57.712162 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.712082 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/kube-multus-additional-cni-plugins/0.log" Jul 06 23:04:57.733762 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.733736 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/egress-router-binary-copy/0.log" Jul 06 23:04:57.755412 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.755383 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/cni-plugins/0.log" Jul 06 23:04:57.777386 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.777361 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/bond-cni-plugin/0.log" Jul 06 23:04:57.801594 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.801566 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/routeoverride-cni/0.log" Jul 06 23:04:57.824628 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.824603 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/whereabouts-cni-bincopy/0.log" Jul 06 23:04:57.848553 ip-10-0-135-55 kubenswrapper[2564]: I0706 23:04:57.848523 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-zkjhk_3c00558f-c19d-4d65-82c8-73dd2d6d094e/whereabouts-cni/0.log"