Jul 10 13:08:05.028336 ip-10-0-140-11 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 13:08:05.028348 ip-10-0-140-11 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 13:08:05.028357 ip-10-0-140-11 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 13:08:05.028664 ip-10-0-140-11 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 13:08:15.055272 ip-10-0-140-11 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 13:08:15.055288 ip-10-0-140-11 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 95f7a4ba179e4f37811b4d41fcdcbdcf -- Jul 10 13:10:19.153116 ip-10-0-140-11 systemd[1]: Starting Kubernetes Kubelet... Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.615270 2567 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620899 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620911 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620914 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620917 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620921 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620924 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620927 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:19.644644 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620931 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620933 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620936 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620939 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620941 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620944 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620947 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620949 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620952 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620955 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620957 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620959 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620962 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620965 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620967 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620970 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620974 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620978 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620981 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:19.646156 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620984 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620987 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620990 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620993 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620995 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.620998 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621000 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621003 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621005 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621008 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621011 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621013 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621015 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621018 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621022 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621026 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621029 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621038 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621041 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:19.647114 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621043 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621046 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621049 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621051 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621054 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621056 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621059 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621062 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621065 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621068 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621070 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621072 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621075 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621078 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621080 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621082 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621085 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621087 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621090 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621093 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:19.647790 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621095 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621098 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621100 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621103 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621105 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621108 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621110 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621113 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621116 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621118 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621121 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621124 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621126 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621129 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621131 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621134 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621136 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621139 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621141 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:19.648779 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621145 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621148 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621150 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621153 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621155 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621158 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621160 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621163 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621167 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621171 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621174 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621177 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621179 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621182 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621184 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621858 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621865 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621868 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621870 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:19.649460 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621873 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621876 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621879 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621882 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621885 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621888 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621891 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621893 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621896 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621899 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621901 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621904 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621906 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621909 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621912 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621914 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621917 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621919 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621922 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621924 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:19.650372 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621927 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621929 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621932 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621934 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621936 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621939 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621942 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621945 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621947 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621949 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621952 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621954 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621957 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621960 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621962 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621965 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621968 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621972 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621975 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:19.655174 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621977 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621980 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621983 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621985 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621988 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621991 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621994 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.621998 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622003 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622006 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622009 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622012 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622015 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622017 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622020 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622023 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622026 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622028 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622031 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622033 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:19.655850 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622036 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622038 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622041 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622043 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622046 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622048 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622051 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622054 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622057 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622062 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622066 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622068 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622071 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622074 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622076 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622079 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622083 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622087 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622090 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:19.756327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622092 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:19.709136 ip-10-0-140-11 systemd[1]: Started Kubernetes Kubelet. Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622095 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622098 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622100 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622102 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622105 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622108 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622111 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622113 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622116 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622118 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622121 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622123 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622126 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622128 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622131 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622133 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622499 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622505 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622509 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:19.757098 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622513 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622517 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622520 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622524 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622527 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622530 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622533 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622535 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622538 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622541 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622544 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622546 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622550 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622552 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622556 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622558 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622561 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622564 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622566 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622569 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:19.758004 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622571 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622573 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622576 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622578 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622583 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622586 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622590 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622593 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622595 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622598 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622603 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622605 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622608 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622611 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622613 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622616 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622618 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622621 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622623 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:19.758836 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622626 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622628 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622631 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622633 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622636 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622640 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622642 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622645 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622647 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622650 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622653 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622655 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622658 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622660 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622662 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622665 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622668 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622670 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622673 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:19.759505 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622675 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622677 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622680 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622682 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622687 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622689 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622692 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622694 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622697 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622699 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622702 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622704 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622708 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622712 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622715 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622717 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622720 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622722 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622726 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622728 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:19.760415 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622731 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622733 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622735 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622738 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622741 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622744 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622746 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622749 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622752 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622754 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622757 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622759 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622762 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622764 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622766 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622769 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622773 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.622776 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622856 2567 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622863 2567 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622868 2567 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 13:10:19.761088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622872 2567 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622876 2567 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622879 2567 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622887 2567 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622891 2567 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622895 2567 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622897 2567 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622901 2567 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622904 2567 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622908 2567 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622911 2567 flags.go:64] FLAG: --cgroup-root="" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622914 2567 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622917 2567 flags.go:64] FLAG: --client-ca-file="" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622920 2567 flags.go:64] FLAG: --cloud-provider="external" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622923 2567 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622927 2567 flags.go:64] FLAG: --cluster-domain="" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622930 2567 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622933 2567 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622936 2567 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622939 2567 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622943 2567 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622946 2567 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622949 2567 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 13:10:19.761825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622952 2567 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622955 2567 flags.go:64] FLAG: --contention-profiling="false" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622958 2567 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622961 2567 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622966 2567 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622968 2567 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622976 2567 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622979 2567 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622982 2567 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622985 2567 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622988 2567 flags.go:64] FLAG: --enable-server="true" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622991 2567 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622994 2567 flags.go:64] FLAG: --event-burst="100" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.622998 2567 flags.go:64] FLAG: --event-qps="50" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623001 2567 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623004 2567 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623007 2567 flags.go:64] FLAG: --eviction-hard="" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623010 2567 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623014 2567 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623017 2567 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623020 2567 flags.go:64] FLAG: --eviction-soft="" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623023 2567 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623026 2567 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623029 2567 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623031 2567 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 13:10:19.762536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623034 2567 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623037 2567 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623040 2567 flags.go:64] FLAG: --feature-gates="" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623044 2567 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623046 2567 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623049 2567 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623053 2567 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623056 2567 flags.go:64] FLAG: --healthz-port="10248" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623058 2567 flags.go:64] FLAG: --help="false" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623061 2567 flags.go:64] FLAG: --hostname-override="ip-10-0-140-11.ec2.internal" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623064 2567 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623068 2567 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623071 2567 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623074 2567 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623078 2567 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623080 2567 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623083 2567 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623086 2567 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623088 2567 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623091 2567 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623094 2567 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623097 2567 flags.go:64] FLAG: --kube-reserved="" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623100 2567 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623103 2567 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 13:10:19.763407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623106 2567 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623110 2567 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623113 2567 flags.go:64] FLAG: --lock-file="" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623115 2567 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623118 2567 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623121 2567 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623126 2567 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623129 2567 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623131 2567 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623134 2567 flags.go:64] FLAG: --logging-format="text" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623137 2567 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623140 2567 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623143 2567 flags.go:64] FLAG: --manifest-url="" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623146 2567 flags.go:64] FLAG: --manifest-url-header="" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623150 2567 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623153 2567 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623156 2567 flags.go:64] FLAG: --max-pods="110" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623159 2567 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623162 2567 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623166 2567 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623169 2567 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623171 2567 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623174 2567 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623177 2567 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623183 2567 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 13:10:19.764114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623187 2567 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623190 2567 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623192 2567 flags.go:64] FLAG: --pod-cidr="" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623195 2567 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623201 2567 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623204 2567 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623207 2567 flags.go:64] FLAG: --pods-per-core="0" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623210 2567 flags.go:64] FLAG: --port="10250" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623214 2567 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623217 2567 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-092b18071470161d6" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623220 2567 flags.go:64] FLAG: --qos-reserved="" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623223 2567 flags.go:64] FLAG: --read-only-port="10255" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623226 2567 flags.go:64] FLAG: --register-node="true" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623228 2567 flags.go:64] FLAG: --register-with-taints="" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623232 2567 flags.go:64] FLAG: --registry-burst="10" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623235 2567 flags.go:64] FLAG: --registry-qps="5" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623237 2567 flags.go:64] FLAG: --reserved-cpus="" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623240 2567 flags.go:64] FLAG: --reserved-memory="" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623243 2567 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623246 2567 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623249 2567 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623252 2567 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623255 2567 flags.go:64] FLAG: --runonce="false" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623257 2567 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623260 2567 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 13:10:19.764940 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623263 2567 flags.go:64] FLAG: --seccomp-default="false" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623266 2567 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623272 2567 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623275 2567 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623278 2567 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623280 2567 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623283 2567 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623286 2567 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623288 2567 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623291 2567 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623294 2567 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623297 2567 flags.go:64] FLAG: --system-cgroups="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623300 2567 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623305 2567 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623308 2567 flags.go:64] FLAG: --tls-cert-file="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623311 2567 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623319 2567 flags.go:64] FLAG: --tls-min-version="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623322 2567 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623325 2567 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623327 2567 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623330 2567 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623333 2567 flags.go:64] FLAG: --v="2" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623337 2567 flags.go:64] FLAG: --version="false" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623340 2567 flags.go:64] FLAG: --vmodule="" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623344 2567 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 13:10:19.765751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.623347 2567 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623451 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623455 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623458 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623461 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623463 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623466 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623469 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623471 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623475 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623478 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623480 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623483 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623485 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623488 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623490 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623493 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623495 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623498 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623501 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:19.767050 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623503 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623506 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623510 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623512 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623515 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623517 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623520 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623522 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623525 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623527 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623531 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623534 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623536 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623539 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623541 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623544 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623547 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623551 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623554 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:19.767712 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623556 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623559 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623564 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623566 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623569 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623571 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623574 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623576 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623579 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623582 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623584 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623587 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623589 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623592 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623594 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623599 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623602 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623604 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623607 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623609 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:19.768327 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623612 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623615 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623617 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623619 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623622 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623624 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623627 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623629 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623632 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623634 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623637 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623639 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623641 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623644 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623647 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623650 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623652 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623655 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623657 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623660 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:19.768901 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623663 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623665 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623669 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623672 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623674 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623678 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623681 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623688 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623691 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623695 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623697 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623700 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623703 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623706 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623708 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623711 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623714 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623716 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623719 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:19.769531 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623721 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.623724 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.624792 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.636636 2567 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.636650 2567 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.636668 2567 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.636674 2567 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636729 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636736 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636740 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636743 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636747 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636751 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636754 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:19.770161 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636757 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636759 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636762 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636765 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636767 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636770 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636775 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636778 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636780 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636783 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636785 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636788 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636790 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636793 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636796 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636798 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636801 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636803 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636806 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636808 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:19.770625 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636811 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636814 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636817 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636819 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636822 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636825 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636829 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636832 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636834 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636837 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636839 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636862 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636865 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636867 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636872 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636876 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636879 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636881 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636886 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:19.771556 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636891 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636894 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636897 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636900 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636903 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636906 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636908 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636911 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636914 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636916 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636919 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636922 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636925 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636927 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636930 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636932 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636935 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636937 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636940 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636944 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:19.772391 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636947 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636950 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636952 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636955 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636957 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636959 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636962 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636964 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636967 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636969 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636972 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636977 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636979 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636982 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636984 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636987 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636989 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636992 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636994 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636997 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:19.942249 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.636999 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637002 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637004 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637007 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637009 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637012 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637015 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637017 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637019 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637022 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637024 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637028 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637031 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.637036 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637146 2567 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 13:10:19.959856 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637152 2567 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637156 2567 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637159 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637163 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637166 2567 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637169 2567 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637171 2567 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637174 2567 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637178 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637181 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637183 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637186 2567 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637189 2567 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637191 2567 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637194 2567 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637196 2567 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637199 2567 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637201 2567 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637204 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637207 2567 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 13:10:19.960442 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637209 2567 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637212 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637214 2567 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637217 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637219 2567 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637222 2567 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637224 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637227 2567 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637229 2567 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637233 2567 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637235 2567 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637238 2567 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637240 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637242 2567 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637245 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637248 2567 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637250 2567 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637254 2567 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637258 2567 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 13:10:19.961088 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637261 2567 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637266 2567 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637269 2567 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637271 2567 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637274 2567 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637278 2567 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637281 2567 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637283 2567 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637286 2567 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637288 2567 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637291 2567 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637293 2567 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637296 2567 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637298 2567 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637301 2567 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637303 2567 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637306 2567 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637308 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637310 2567 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 13:10:19.961648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637313 2567 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637315 2567 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637318 2567 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637322 2567 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637325 2567 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637327 2567 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637330 2567 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637332 2567 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637335 2567 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637337 2567 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637340 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637342 2567 feature_gate.go:328] unrecognized feature gate: Example Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637345 2567 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637347 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637351 2567 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637354 2567 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637357 2567 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637359 2567 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637362 2567 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 13:10:19.962292 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637366 2567 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637368 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637370 2567 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637373 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637375 2567 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637378 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637380 2567 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637383 2567 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637385 2567 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637388 2567 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637390 2567 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637393 2567 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637395 2567 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637398 2567 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637401 2567 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637403 2567 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637407 2567 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637410 2567 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637413 2567 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637415 2567 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 13:10:19.962866 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:19.637418 2567 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.637422 2567 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.637513 2567 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.642862 2567 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.644713 2567 server.go:1019] "Starting client certificate rotation" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.647716 2567 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.650961 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.650985 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.655393 2567 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.666412 2567 log.go:25] "Validated CRI v1 runtime API" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.675012 2567 log.go:25] "Validated CRI v1 image API" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.675814 2567 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 13:10:19.963533 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.678649 2567 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e16dd6ef-5ed7-4069-9386-e022b224dd6b:/dev/nvme0n1p4 e8cb8a56-ae58-4e32-9594-a65d6b9d462a:/dev/nvme0n1p3] Jul 10 13:10:19.963973 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.678669 2567 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.685024 2567 manager.go:216] Machine: {Timestamp:2026-07-10 13:10:19.684513354 +0000 UTC m=+0.400890974 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099858 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22010b02bd6ce37402f7036f498a99 SystemUUID:ec22010b-02bd-6ce3-7402-f7036f498a99 BootID:95f7a4ba-179e-4f37-811b-4d41fcdcbdcf Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e8:1f:18:8b:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e8:1f:18:8b:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:64:0c:ad:7b:dc Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.685124 2567 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.685235 2567 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.687353 2567 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.687398 2567 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.688397 2567 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.688407 2567 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 13:10:19.964099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.688415 2567 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.690238 2567 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.691400 2567 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.693424 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.695992 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696114 2567 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696144 2567 kubelet.go:486] "Attempting to sync node with API server" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696156 2567 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696180 2567 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696194 2567 kubelet.go:398] "Adding apiserver pod source" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696204 2567 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696612 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.696632 2567 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.699939 2567 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.701736 2567 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.701766 2567 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702561 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702577 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702583 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702588 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702597 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702603 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702712 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702718 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702729 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702736 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702748 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.702800 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.704745 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.705006 2567 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.708167 2567 server.go:1268] "Started kubelet" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.708487 2567 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.708389 2567 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.708705 2567 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.709186 2567 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.710270 2567 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.711239 2567 server.go:310] "Adding debug handlers to kubelet server" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.714381 2567 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.715697 2567 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.716730 2567 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.716843 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717346 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-prmpk" Jul 10 13:10:19.964408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717506 2567 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717549 2567 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717570 2567 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717676 2567 reconstruct.go:97] "Volume reconstruction finished" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717684 2567 factory.go:55] Registering systemd factory Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717686 2567 reconciler.go:29] "Reconciler: start to sync state" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717708 2567 factory.go:223] Registration of the systemd container factory successfully Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717982 2567 factory.go:153] Registering CRI-O factory Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.717993 2567 factory.go:223] Registration of the crio container factory successfully Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.718055 2567 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.718080 2567 factory.go:103] Registering Raw factory Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.718150 2567 manager.go:1196] Started watching for new ooms in manager Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.718619 2567 manager.go:349] Starting recovery of all containers Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.720956 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.729351 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-prmpk" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.731696 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.731820 2567 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.736351 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.736438 2567 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-11.ec2.internal.18c0efb68977ed7f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-11.ec2.internal,UID:ip-10-0-140-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-11.ec2.internal,},FirstTimestamp:2026-07-10 13:10:19.708140927 +0000 UTC m=+0.424518548,LastTimestamp:2026-07-10 13:10:19.708140927 +0000 UTC m=+0.424518548,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-11.ec2.internal,}" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.746797 2567 manager.go:354] Recovery completed Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.751305 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753047 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753072 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753083 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753528 2567 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753537 2567 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753556 2567 state_mem.go:36] "Initialized new in-memory state store" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.753539 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.754231 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.754258 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:19.965688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.754271 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.754836 2567 policy_none.go:49] "None policy: Start" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.754850 2567 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.754861 2567 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.761738 2567 policy_none.go:47] "Start" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.816450 2567 manager.go:337] "Starting Device Plugin manager" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.816612 2567 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.816623 2567 server.go:87] "Starting device plugin registration server" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.816816 2567 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.816827 2567 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.816967 2567 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.817059 2567 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.817066 2567 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.817595 2567 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.817653 2567 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.917539 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.918653 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.918686 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.918698 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.918723 2567 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-11.ec2.internal" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.928127 2567 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-11.ec2.internal" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.928145 2567 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-11.ec2.internal\": node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.943126 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.944515 2567 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.944552 2567 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.944576 2567 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.944616 2567 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:19.945313 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:19.966585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:19.946874 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.045865 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.045870 2567 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal"] Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.045932 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.047454 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.047480 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.047507 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049030 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049092 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049118 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049710 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049719 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049737 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049757 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049763 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.049771 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.051154 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.051177 2567 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.052383 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientMemory" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.052405 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.052417 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeHasSufficientPID" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.087541 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-11.ec2.internal\" not found" node="ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.091991 2567 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-11.ec2.internal\" not found" node="ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.118960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b229af0a3dd64d765568e457658e38-config\") pod \"kube-apiserver-proxy-ip-10-0-140-11.ec2.internal\" (UID: \"87b229af0a3dd64d765568e457658e38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.118989 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34ab660bb6c61b2b8d3e0a2570503477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal\" (UID: \"34ab660bb6c61b2b8d3e0a2570503477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.119006 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34ab660bb6c61b2b8d3e0a2570503477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal\" (UID: \"34ab660bb6c61b2b8d3e0a2570503477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.167564 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.146624 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.220088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34ab660bb6c61b2b8d3e0a2570503477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal\" (UID: \"34ab660bb6c61b2b8d3e0a2570503477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.220134 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34ab660bb6c61b2b8d3e0a2570503477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal\" (UID: \"34ab660bb6c61b2b8d3e0a2570503477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.220162 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34ab660bb6c61b2b8d3e0a2570503477-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal\" (UID: \"34ab660bb6c61b2b8d3e0a2570503477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.220183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b229af0a3dd64d765568e457658e38-config\") pod \"kube-apiserver-proxy-ip-10-0-140-11.ec2.internal\" (UID: \"87b229af0a3dd64d765568e457658e38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.220183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/34ab660bb6c61b2b8d3e0a2570503477-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal\" (UID: \"34ab660bb6c61b2b8d3e0a2570503477\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.220210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/87b229af0a3dd64d765568e457658e38-config\") pod \"kube-apiserver-proxy-ip-10-0-140-11.ec2.internal\" (UID: \"87b229af0a3dd64d765568e457658e38\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.340325 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.247641 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.470701 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.349176 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.470701 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.391563 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.470701 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.396197 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:20.470701 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.449502 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.636738 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.551052 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.645287 2567 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.645426 2567 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.651541 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.736360 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 13:05:20 +0000 UTC" deadline="2028-01-15 00:30:47.24514272 +0000 UTC" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.736383 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13283h20m26.508763379s" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:20.748579 2567 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:20.775209 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.752028 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:20.935003 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.852615 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:20.953274 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:21.015083 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b229af0a3dd64d765568e457658e38.slice/crio-392135fbeb691d60dc1cf75a4bbd13c3a9a7279a701b31823bf08cf19105e1f1 WatchSource:0}: Error finding container 392135fbeb691d60dc1cf75a4bbd13c3a9a7279a701b31823bf08cf19105e1f1: Status 404 returned error can't find the container with id 392135fbeb691d60dc1cf75a4bbd13c3a9a7279a701b31823bf08cf19105e1f1 Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:21.021702 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ab660bb6c61b2b8d3e0a2570503477.slice/crio-0423c7c69d521c4ae10762afa69b18501afaa3e86127770a8fb319f7a3cbe76f WatchSource:0}: Error finding container 0423c7c69d521c4ae10762afa69b18501afaa3e86127770a8fb319f7a3cbe76f: Status 404 returned error can't find the container with id 0423c7c69d521c4ae10762afa69b18501afaa3e86127770a8fb319f7a3cbe76f Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.024026 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.053624 2567 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-11.ec2.internal\" not found" Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.092214 2567 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:21.122089 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.117982 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" Jul 10 13:10:21.236336 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.129502 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 13:10:21.236336 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.130903 2567 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" Jul 10 13:10:21.236336 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.142188 2567 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 13:10:21.364159 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.278807 2567 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:21.702779 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.697265 2567 apiserver.go:52] "Watching apiserver" Jul 10 13:10:21.709324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.704894 2567 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 13:10:21.709324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.705247 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-5gmgc","openshift-dns/node-resolver-r4gbb","openshift-image-registry/node-ca-65xfz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal","openshift-multus/network-metrics-daemon-dlr7k","openshift-network-operator/iptables-alerter-8l2wn","kube-system/konnectivity-agent-dc9tb","kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5","openshift-multus/multus-additional-cni-plugins-98dtr","openshift-multus/multus-hldt7","openshift-network-diagnostics/network-check-target-tmb52","openshift-ovn-kubernetes/ovnkube-node-knf4x","kube-system/global-pull-secret-syncer-xhc8c"] Jul 10 13:10:21.709324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.707168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.709324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.709125 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.710734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.710004 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 13:10:21.710734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.710208 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.710734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.710237 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6ww8h\"" Jul 10 13:10:21.710734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.710467 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.710734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.710533 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 13:10:21.710734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.710473 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 13:10:21.715725 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.715187 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 13:10:21.715725 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.715569 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gd88g\"" Jul 10 13:10:21.715873 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.715821 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 13:10:21.716297 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.716039 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.716297 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.716263 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 13:10:21.717191 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.716870 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.717270 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.717241 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 13:10:21.717583 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.717546 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.719032 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.719097 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.719412 2567 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.720223 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.720231 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-m8lbz\"" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.720281 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.720488 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.720665 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.720819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.720807 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.721603 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.720837 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.722216 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.722122 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.727069 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.725397 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.727069 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.725624 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 13:10:21.727069 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.725867 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.730075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.728304 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.730075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.728622 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z79gv\"" Jul 10 13:10:21.730075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.729003 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 13:10:21.730075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.729186 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pnd2n\"" Jul 10 13:10:21.730075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.729423 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.732121 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.731215 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.733974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.733952 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.734756 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.734565 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9lm7b\"" Jul 10 13:10:21.734756 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.734596 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 13:10:21.734756 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.734710 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 13:10:21.736986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.736279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.736986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.736501 2567 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 13:10:21.736986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.736777 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.737416 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.737263 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v494x\"" Jul 10 13:10:21.737416 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.737302 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.737966 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.737603 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.738675 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.738521 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 13:10:21.738943 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.738867 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-chfvv\"" Jul 10 13:10:21.739238 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.739108 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:10:21.739668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.739646 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:21.739734 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.739712 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:21.740270 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.740097 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 13:10:21.740375 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.740357 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lwwxw\"" Jul 10 13:10:21.740852 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.740834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-kubelet-config\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.740911 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.740871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-system-cni-dir\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.740911 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.740899 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cnibin\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.741034 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741016 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-kubelet\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741081 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741050 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-node-log\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741077 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dv96\" (UniqueName: \"kubernetes.io/projected/09abb34b-0b90-4794-9ff3-87a3f23fed12-kube-api-access-6dv96\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhtz\" (UniqueName: \"kubernetes.io/projected/3494da59-74b5-4f5d-83dd-b9ae69096f05-kube-api-access-ljhtz\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.741213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-env-overrides\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741160 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.741213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.741343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-var-lib-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741238 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741263 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-ovn\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-log-socket\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741311 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-dbus\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.741343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741336 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-cni-netd\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741384 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-host\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741441 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741484 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjgs\" (UniqueName: \"kubernetes.io/projected/da325937-f2c1-4ffe-ade9-580cd62f69ef-kube-api-access-wqjgs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-run-ovn-kubernetes\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741589 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovnkube-config\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741614 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-etc-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741654 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovn-node-metrics-cert\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741678 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4ld\" (UniqueName: \"kubernetes.io/projected/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-kube-api-access-vj4ld\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741728 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3494da59-74b5-4f5d-83dd-b9ae69096f05-iptables-alerter-script\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741752 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlf59\" (UniqueName: \"kubernetes.io/projected/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-kube-api-access-hlf59\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741776 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-slash\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741800 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-systemd\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovnkube-script-lib\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741882 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3494da59-74b5-4f5d-83dd-b9ae69096f05-host-slash\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.741908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741920 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-os-release\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.742377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741950 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.742377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.741975 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-systemd-units\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.742377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.742005 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-run-netns\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.742377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.742030 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-cni-bin\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.742377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.742055 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-serviceca\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.759446 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.759427 2567 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dn8x2" Jul 10 13:10:21.771677 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.771629 2567 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dn8x2" Jul 10 13:10:21.818762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.818664 2567 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 13:10:21.842885 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.842845 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:21.842961 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.842901 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-os-release\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.842961 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.842935 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.843075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.842961 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysctl-conf\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.843075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.842983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-registration-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.843075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843005 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-lib-modules\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.843075 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.843026 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:21.843075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843032 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3494da59-74b5-4f5d-83dd-b9ae69096f05-iptables-alerter-script\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.843075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843058 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-socket-dir-parent\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.843106 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:10:22.343070893 +0000 UTC m=+3.059448517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843133 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-sys-fs\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843137 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7j6l4\" (UniqueName: \"kubernetes.io/projected/13881716-db9c-451b-a633-763d9bbfc460-kube-api-access-7j6l4\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843195 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysconfig\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843218 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-run\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843250 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-var-lib-kubelet\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843278 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovnkube-script-lib\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843301 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-socket-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.843333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843325 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-systemd\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.843755 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5887\" (UniqueName: \"kubernetes.io/projected/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-kube-api-access-f5887\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.844096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843967 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovnkube-script-lib\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.843958 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3494da59-74b5-4f5d-83dd-b9ae69096f05-iptables-alerter-script\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.844096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844008 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-os-release\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.844096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844047 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-systemd-units\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844098 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-run-netns\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844129 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-os-release\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-cni-bin\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844178 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-run-netns\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhtz\" (UniqueName: \"kubernetes.io/projected/3494da59-74b5-4f5d-83dd-b9ae69096f05-kube-api-access-ljhtz\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844229 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-tmp-dir\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844243 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-systemd-units\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844301 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-cni-bin\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844257 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-tmp\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844736 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cnibin\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dv96\" (UniqueName: \"kubernetes.io/projected/09abb34b-0b90-4794-9ff3-87a3f23fed12-kube-api-access-6dv96\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844790 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-daemon-config\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844817 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6k2j\" (UniqueName: \"kubernetes.io/projected/30953d48-7c6d-4251-a21d-08c41361d4ba-kube-api-access-w6k2j\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844843 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-system-cni-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.844880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844870 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-env-overrides\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844939 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-var-lib-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.844990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845013 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-ovn\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845036 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-dbus\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845085 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-host\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cnibin\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845112 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-k8s-cni-cncf-io\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845149 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-cni-multus\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-hosts-file\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845200 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-kubernetes\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845228 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-sys\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.845255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845250 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-tuned\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845277 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-run-ovn-kubernetes\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845303 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-multus-certs\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845328 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysctl-d\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845377 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-kubelet\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845404 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovnkube-config\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-etc-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845459 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovn-node-metrics-cert\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845495 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4ld\" (UniqueName: \"kubernetes.io/projected/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-kube-api-access-vj4ld\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845521 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845545 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-cni-bin\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845569 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-hostroot\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlf59\" (UniqueName: \"kubernetes.io/projected/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-kube-api-access-hlf59\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.846033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845621 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-slash\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845647 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-systemd\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845673 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3494da59-74b5-4f5d-83dd-b9ae69096f05-host-slash\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00558e71-8d16-44ce-aa14-d69cdf75fb72-konnectivity-ca\") pod \"konnectivity-agent-dc9tb\" (UID: \"00558e71-8d16-44ce-aa14-d69cdf75fb72\") " pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-cni-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845749 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-netns\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845777 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845802 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-serviceca\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845827 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-kubelet-config\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845850 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-cnibin\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845880 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-kubelet-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-device-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845947 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30953d48-7c6d-4251-a21d-08c41361d4ba-cni-binary-copy\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845974 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-system-cni-dir\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.846743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.845999 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-kubelet\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-node-log\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846047 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00558e71-8d16-44ce-aa14-d69cdf75fb72-agent-certs\") pod \"konnectivity-agent-dc9tb\" (UID: \"00558e71-8d16-44ce-aa14-d69cdf75fb72\") " pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846073 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-etc-kubernetes\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846099 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvmz\" (UniqueName: \"kubernetes.io/projected/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-kube-api-access-6hvmz\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846124 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-modprobe-d\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846160 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjgs\" (UniqueName: \"kubernetes.io/projected/da325937-f2c1-4ffe-ade9-580cd62f69ef-kube-api-access-wqjgs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-log-socket\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846213 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-conf-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846236 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-etc-selinux\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846260 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-host\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846286 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846311 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-cni-netd\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.846347 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:21.847461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846391 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-cni-netd\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.846412 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:22.346395178 +0000 UTC m=+3.062772789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846509 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846523 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-etc-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846549 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-host\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846594 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-ovn\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-system-cni-dir\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846692 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-kubelet\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846727 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-slash\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846737 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-run-systemd\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.846792 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3494da59-74b5-4f5d-83dd-b9ae69096f05-host-slash\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847009 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-log-socket\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847056 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-host-run-ovn-kubernetes\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847325 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.848250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847414 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-env-overrides\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847740 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-serviceca\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847817 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-kubelet-config\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.847866 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-node-log\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.848029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovnkube-config\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.848352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.848491 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/09abb34b-0b90-4794-9ff3-87a3f23fed12-var-lib-openvswitch\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.848639 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-dbus\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:21.848967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.848732 2567 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 13:10:21.849434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.849101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.854649 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.854285 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dv96\" (UniqueName: \"kubernetes.io/projected/09abb34b-0b90-4794-9ff3-87a3f23fed12-kube-api-access-6dv96\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.854649 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.854552 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/09abb34b-0b90-4794-9ff3-87a3f23fed12-ovn-node-metrics-cert\") pod \"ovnkube-node-knf4x\" (UID: \"09abb34b-0b90-4794-9ff3-87a3f23fed12\") " pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:21.855668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.855640 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhtz\" (UniqueName: \"kubernetes.io/projected/3494da59-74b5-4f5d-83dd-b9ae69096f05-kube-api-access-ljhtz\") pod \"iptables-alerter-8l2wn\" (UID: \"3494da59-74b5-4f5d-83dd-b9ae69096f05\") " pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:21.860703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.856610 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjgs\" (UniqueName: \"kubernetes.io/projected/da325937-f2c1-4ffe-ade9-580cd62f69ef-kube-api-access-wqjgs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:21.860703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.858698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlf59\" (UniqueName: \"kubernetes.io/projected/d9700277-f8d3-4ca9-8da5-7f728bfbd9a5-kube-api-access-hlf59\") pod \"multus-additional-cni-plugins-98dtr\" (UID: \"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5\") " pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:21.860703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.858837 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4ld\" (UniqueName: \"kubernetes.io/projected/f6a6b054-52ca-4dfe-abf5-2ef50b8a3226-kube-api-access-vj4ld\") pod \"node-ca-65xfz\" (UID: \"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226\") " pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:21.946789 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-multus-certs\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946798 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysctl-d\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946825 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946848 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-kubelet\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946898 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-multus-certs\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946900 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-cni-bin\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946938 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-cni-bin\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.946959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946958 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-hostroot\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.946992 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00558e71-8d16-44ce-aa14-d69cdf75fb72-konnectivity-ca\") pod \"konnectivity-agent-dc9tb\" (UID: \"00558e71-8d16-44ce-aa14-d69cdf75fb72\") " pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947018 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-cni-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947044 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-netns\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysctl-d\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-cnibin\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947101 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-kubelet-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947120 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-device-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30953d48-7c6d-4251-a21d-08c41361d4ba-cni-binary-copy\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947182 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00558e71-8d16-44ce-aa14-d69cdf75fb72-agent-certs\") pod \"konnectivity-agent-dc9tb\" (UID: \"00558e71-8d16-44ce-aa14-d69cdf75fb72\") " pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947206 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-etc-kubernetes\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947231 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvmz\" (UniqueName: \"kubernetes.io/projected/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-kube-api-access-6hvmz\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947255 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-modprobe-d\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947306 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-kubelet\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.947329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947321 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-conf-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947344 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-etc-selinux\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947378 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-host\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-os-release\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947448 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-device-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947457 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysctl-conf\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947488 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-registration-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947507 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-hostroot\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947512 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-lib-modules\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-socket-dir-parent\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947579 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-sys-fs\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947605 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7j6l4\" (UniqueName: \"kubernetes.io/projected/13881716-db9c-451b-a633-763d9bbfc460-kube-api-access-7j6l4\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947627 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-conf-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947628 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-lib-modules\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947728 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-etc-selinux\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947873 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-modprobe-d\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysconfig\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948029 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysctl-conf\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948070 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-netns\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948079 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-host\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948090 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-cnibin\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.947631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-sysconfig\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948139 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-os-release\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948156 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-sys-fs\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948186 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-run\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-registration-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948210 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-socket-dir-parent\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948140 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-run\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948227 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-kubelet-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.948732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-var-lib-kubelet\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948304 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-var-lib-kubelet\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948314 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/00558e71-8d16-44ce-aa14-d69cdf75fb72-konnectivity-ca\") pod \"konnectivity-agent-dc9tb\" (UID: \"00558e71-8d16-44ce-aa14-d69cdf75fb72\") " pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948335 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-socket-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948373 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-systemd\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5887\" (UniqueName: \"kubernetes.io/projected/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-kube-api-access-f5887\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948421 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13881716-db9c-451b-a633-763d9bbfc460-socket-dir\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948444 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-etc-kubernetes\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948429 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-tmp-dir\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-systemd\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948528 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-tmp\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948574 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-daemon-config\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-cni-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948603 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6k2j\" (UniqueName: \"kubernetes.io/projected/30953d48-7c6d-4251-a21d-08c41361d4ba-kube-api-access-w6k2j\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948630 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-system-cni-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.949496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948674 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-k8s-cni-cncf-io\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948703 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-cni-multus\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948713 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-tmp-dir\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948688 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-system-cni-dir\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948732 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-hosts-file\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948762 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-kubernetes\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948778 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30953d48-7c6d-4251-a21d-08c41361d4ba-cni-binary-copy\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-sys\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-tuned\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.948970 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-var-lib-cni-multus\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.949017 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-hosts-file\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.949074 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-kubernetes\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.949114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-sys\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.949101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/30953d48-7c6d-4251-a21d-08c41361d4ba-host-run-k8s-cni-cncf-io\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.950252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.949146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/30953d48-7c6d-4251-a21d-08c41361d4ba-multus-daemon-config\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.951690 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.951665 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-etc-tuned\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.951823 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.951732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/00558e71-8d16-44ce-aa14-d69cdf75fb72-agent-certs\") pod \"konnectivity-agent-dc9tb\" (UID: \"00558e71-8d16-44ce-aa14-d69cdf75fb72\") " pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:21.953690 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.953612 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-tmp\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.955032 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.954959 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:21.955032 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.954986 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:21.955032 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.955000 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:21.955313 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:21.955078 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:22.455058972 +0000 UTC m=+3.171436596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:21.956127 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.956064 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" event={"ID":"34ab660bb6c61b2b8d3e0a2570503477","Type":"ContainerStarted","Data":"0423c7c69d521c4ae10762afa69b18501afaa3e86127770a8fb319f7a3cbe76f"} Jul 10 13:10:21.957850 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.957787 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" event={"ID":"87b229af0a3dd64d765568e457658e38","Type":"ContainerStarted","Data":"392135fbeb691d60dc1cf75a4bbd13c3a9a7279a701b31823bf08cf19105e1f1"} Jul 10 13:10:21.958320 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.958300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvmz\" (UniqueName: \"kubernetes.io/projected/615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde-kube-api-access-6hvmz\") pod \"node-resolver-r4gbb\" (UID: \"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde\") " pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:21.962032 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.961851 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6k2j\" (UniqueName: \"kubernetes.io/projected/30953d48-7c6d-4251-a21d-08c41361d4ba-kube-api-access-w6k2j\") pod \"multus-hldt7\" (UID: \"30953d48-7c6d-4251-a21d-08c41361d4ba\") " pod="openshift-multus/multus-hldt7" Jul 10 13:10:21.962140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.962040 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5887\" (UniqueName: \"kubernetes.io/projected/b3eb08a4-b6f2-4d58-a93c-c0b962e5d365-kube-api-access-f5887\") pod \"tuned-5gmgc\" (UID: \"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365\") " pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:21.963794 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:21.963766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7j6l4\" (UniqueName: \"kubernetes.io/projected/13881716-db9c-451b-a633-763d9bbfc460-kube-api-access-7j6l4\") pod \"aws-ebs-csi-driver-node-99wz5\" (UID: \"13881716-db9c-451b-a633-763d9bbfc460\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:22.028197 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.028116 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-98dtr" Jul 10 13:10:22.038348 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.038317 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9700277_f8d3_4ca9_8da5_7f728bfbd9a5.slice/crio-1d351e14896b301705d49991941e74483b7b46640a223f51d059275aadb027bf WatchSource:0}: Error finding container 1d351e14896b301705d49991941e74483b7b46640a223f51d059275aadb027bf: Status 404 returned error can't find the container with id 1d351e14896b301705d49991941e74483b7b46640a223f51d059275aadb027bf Jul 10 13:10:22.039028 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.039008 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:22.046512 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.046491 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09abb34b_0b90_4794_9ff3_87a3f23fed12.slice/crio-12f76d0fe4a2cb6219045dd899e7d9d2eeab3248f9837dfb787afb26275321cb WatchSource:0}: Error finding container 12f76d0fe4a2cb6219045dd899e7d9d2eeab3248f9837dfb787afb26275321cb: Status 404 returned error can't find the container with id 12f76d0fe4a2cb6219045dd899e7d9d2eeab3248f9837dfb787afb26275321cb Jul 10 13:10:22.051403 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.051382 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-65xfz" Jul 10 13:10:22.058448 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.058426 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6a6b054_52ca_4dfe_abf5_2ef50b8a3226.slice/crio-4c1f0a67efbba5e2202a9e7eb70b173ece78db3c53af60e5117b691ee9ef95e6 WatchSource:0}: Error finding container 4c1f0a67efbba5e2202a9e7eb70b173ece78db3c53af60e5117b691ee9ef95e6: Status 404 returned error can't find the container with id 4c1f0a67efbba5e2202a9e7eb70b173ece78db3c53af60e5117b691ee9ef95e6 Jul 10 13:10:22.076243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.076223 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8l2wn" Jul 10 13:10:22.083182 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.083162 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3494da59_74b5_4f5d_83dd_b9ae69096f05.slice/crio-00692596ef7f1a0ce32f0ca6bfbdb5d1121ebf355987e6553d8a327e917bbe2a WatchSource:0}: Error finding container 00692596ef7f1a0ce32f0ca6bfbdb5d1121ebf355987e6553d8a327e917bbe2a: Status 404 returned error can't find the container with id 00692596ef7f1a0ce32f0ca6bfbdb5d1121ebf355987e6553d8a327e917bbe2a Jul 10 13:10:22.086450 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.086434 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" Jul 10 13:10:22.093888 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.093860 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13881716_db9c_451b_a633_763d9bbfc460.slice/crio-8bb1d0e0e479123d60938a5d6437b07044da76509cda4c783f308c2c54a3f618 WatchSource:0}: Error finding container 8bb1d0e0e479123d60938a5d6437b07044da76509cda4c783f308c2c54a3f618: Status 404 returned error can't find the container with id 8bb1d0e0e479123d60938a5d6437b07044da76509cda4c783f308c2c54a3f618 Jul 10 13:10:22.095996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.095962 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:22.104890 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.104838 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-r4gbb" Jul 10 13:10:22.105813 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.105789 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00558e71_8d16_44ce_aa14_d69cdf75fb72.slice/crio-45cd8d7e2cbff8eb5aaa388d958a9776379dd057835664b9709e5ee2a0421dfa WatchSource:0}: Error finding container 45cd8d7e2cbff8eb5aaa388d958a9776379dd057835664b9709e5ee2a0421dfa: Status 404 returned error can't find the container with id 45cd8d7e2cbff8eb5aaa388d958a9776379dd057835664b9709e5ee2a0421dfa Jul 10 13:10:22.112378 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.112357 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" Jul 10 13:10:22.113503 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.113478 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615c6ac8_7da2_4ea6_9c80_4a8e5f7d0fde.slice/crio-6e39665bc3ee93099856a2379e95ae1c1fa8d7fb159a989772abcfa712fefd51 WatchSource:0}: Error finding container 6e39665bc3ee93099856a2379e95ae1c1fa8d7fb159a989772abcfa712fefd51: Status 404 returned error can't find the container with id 6e39665bc3ee93099856a2379e95ae1c1fa8d7fb159a989772abcfa712fefd51 Jul 10 13:10:22.120193 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.120045 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hldt7" Jul 10 13:10:22.120953 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.120927 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3eb08a4_b6f2_4d58_a93c_c0b962e5d365.slice/crio-97cca79b868d511b02bb3e7fa94384e1cd72790c6d5798b2195f04ecb7c3c900 WatchSource:0}: Error finding container 97cca79b868d511b02bb3e7fa94384e1cd72790c6d5798b2195f04ecb7c3c900: Status 404 returned error can't find the container with id 97cca79b868d511b02bb3e7fa94384e1cd72790c6d5798b2195f04ecb7c3c900 Jul 10 13:10:22.129592 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:10:22.129564 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30953d48_7c6d_4251_a21d_08c41361d4ba.slice/crio-021b3f9609a56b3e2c5804a2c09ebbfa4924fe39ece1c938040d1dc1901a6712 WatchSource:0}: Error finding container 021b3f9609a56b3e2c5804a2c09ebbfa4924fe39ece1c938040d1dc1901a6712: Status 404 returned error can't find the container with id 021b3f9609a56b3e2c5804a2c09ebbfa4924fe39ece1c938040d1dc1901a6712 Jul 10 13:10:22.206637 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.206612 2567 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 13:10:22.352626 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.352543 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:22.352626 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.352612 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:22.352823 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.352689 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:22.352823 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.352766 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:10:23.352746652 +0000 UTC m=+4.069124286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:22.352823 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.352692 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:22.352985 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.352831 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:23.352819129 +0000 UTC m=+4.069196736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:22.553524 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.553490 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:22.553721 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.553657 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:22.553721 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.553675 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:22.553721 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.553684 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:22.553903 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.553744 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:23.553727108 +0000 UTC m=+4.270104729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:22.774764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.774718 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 13:05:22 +0000 UTC" deadline="2028-02-29 12:48:40.924913507 +0000 UTC" Jul 10 13:10:22.774764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.774769 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14375h38m18.150162707s" Jul 10 13:10:22.945622 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.945570 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:22.946101 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.945570 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:22.946101 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.945703 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:22.946101 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:22.945795 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:22.961267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.961138 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" event={"ID":"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365","Type":"ContainerStarted","Data":"97cca79b868d511b02bb3e7fa94384e1cd72790c6d5798b2195f04ecb7c3c900"} Jul 10 13:10:22.964972 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.964940 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r4gbb" event={"ID":"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde","Type":"ContainerStarted","Data":"6e39665bc3ee93099856a2379e95ae1c1fa8d7fb159a989772abcfa712fefd51"} Jul 10 13:10:22.966126 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.966099 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dc9tb" event={"ID":"00558e71-8d16-44ce-aa14-d69cdf75fb72","Type":"ContainerStarted","Data":"45cd8d7e2cbff8eb5aaa388d958a9776379dd057835664b9709e5ee2a0421dfa"} Jul 10 13:10:22.967192 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.967165 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" event={"ID":"13881716-db9c-451b-a633-763d9bbfc460","Type":"ContainerStarted","Data":"8bb1d0e0e479123d60938a5d6437b07044da76509cda4c783f308c2c54a3f618"} Jul 10 13:10:22.968130 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.968103 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8l2wn" event={"ID":"3494da59-74b5-4f5d-83dd-b9ae69096f05","Type":"ContainerStarted","Data":"00692596ef7f1a0ce32f0ca6bfbdb5d1121ebf355987e6553d8a327e917bbe2a"} Jul 10 13:10:22.969260 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.969240 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerStarted","Data":"1d351e14896b301705d49991941e74483b7b46640a223f51d059275aadb027bf"} Jul 10 13:10:22.970310 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.970290 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hldt7" event={"ID":"30953d48-7c6d-4251-a21d-08c41361d4ba","Type":"ContainerStarted","Data":"021b3f9609a56b3e2c5804a2c09ebbfa4924fe39ece1c938040d1dc1901a6712"} Jul 10 13:10:22.971440 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.971418 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65xfz" event={"ID":"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226","Type":"ContainerStarted","Data":"4c1f0a67efbba5e2202a9e7eb70b173ece78db3c53af60e5117b691ee9ef95e6"} Jul 10 13:10:22.974143 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:22.974116 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"12f76d0fe4a2cb6219045dd899e7d9d2eeab3248f9837dfb787afb26275321cb"} Jul 10 13:10:23.360806 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.360766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:23.361060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.360835 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:23.361060 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.361003 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:23.361294 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.361070 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:10:25.361049697 +0000 UTC m=+6.077427315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:23.361594 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.361490 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:23.361594 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.361541 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:25.361525729 +0000 UTC m=+6.077903339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:23.563022 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.562366 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:23.563022 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.562552 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:23.563022 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.562572 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:23.563022 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.562586 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:23.563022 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.562645 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:25.562626786 +0000 UTC m=+6.279004399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:23.775881 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.775804 2567 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 13:05:22 +0000 UTC" deadline="2028-03-30 20:31:45.725778084 +0000 UTC" Jul 10 13:10:23.776016 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.775882 2567 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15103h21m21.949900134s" Jul 10 13:10:23.946924 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.946419 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:23.946924 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:23.946553 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:23.982950 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:23.982228 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" event={"ID":"87b229af0a3dd64d765568e457658e38","Type":"ContainerStarted","Data":"7e3c584f15f97e84ceef1d9bf56e957fbdefc702d36990a461419788ef6842c8"} Jul 10 13:10:24.945357 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:24.945323 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:24.945559 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:24.945466 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:24.945559 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:24.945539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:24.945687 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:24.945611 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:25.012888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.012849 2567 generic.go:356] "Generic (PLEG): container finished" podID="34ab660bb6c61b2b8d3e0a2570503477" containerID="8ffdf3c6703183979386cc843dcbeec0b5b05864187449c684f8e91ed10ab716" exitCode=0 Jul 10 13:10:25.013750 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.013647 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" event={"ID":"34ab660bb6c61b2b8d3e0a2570503477","Type":"ContainerDied","Data":"8ffdf3c6703183979386cc843dcbeec0b5b05864187449c684f8e91ed10ab716"} Jul 10 13:10:25.033189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.032866 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-11.ec2.internal" podStartSLOduration=4.032847684 podStartE2EDuration="4.032847684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:23.998959742 +0000 UTC m=+4.715337385" watchObservedRunningTime="2026-07-10 13:10:25.032847684 +0000 UTC m=+5.749225314" Jul 10 13:10:25.383947 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.383059 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:25.383947 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.383131 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:25.383947 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.383298 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:25.383947 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.383359 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:10:29.383338983 +0000 UTC m=+10.099716604 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:25.383947 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.383769 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:25.383947 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.383821 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:29.383804755 +0000 UTC m=+10.100182372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:25.585334 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.585249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:25.585512 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.585489 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:25.585512 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.585509 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:25.585633 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.585523 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:25.585633 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.585579 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:29.585560499 +0000 UTC m=+10.301938121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:25.947789 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:25.947756 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:25.947972 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:25.947900 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:26.023948 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:26.023687 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" event={"ID":"34ab660bb6c61b2b8d3e0a2570503477","Type":"ContainerStarted","Data":"e34fbe50c4feb3b4007454c073b9a04dc37bc733e0758ae6cb080d420433ba78"} Jul 10 13:10:26.945207 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:26.945168 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:26.945207 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:26.945217 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:26.945481 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:26.945324 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:26.945820 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:26.945786 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:27.946265 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:27.946234 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:27.946692 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:27.946373 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:28.944818 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:28.944785 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:28.944991 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:28.944794 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:28.944991 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:28.944912 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:28.945080 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:28.944997 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:29.416701 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:29.416663 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:29.417150 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:29.416735 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:29.417150 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.416895 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:29.417150 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.416958 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:10:37.416940675 +0000 UTC m=+18.133318293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:29.417377 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.417321 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:29.417377 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.417371 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:37.41735721 +0000 UTC m=+18.133734816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:29.618241 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:29.618208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:29.618423 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.618402 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:29.618490 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.618425 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:29.618490 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.618440 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:29.618584 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.618505 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:37.618483851 +0000 UTC m=+18.334861471 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:29.945918 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:29.945890 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:29.946067 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:29.945978 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:30.945105 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:30.945043 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:30.945105 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:30.945066 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:30.945551 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:30.945168 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:30.945551 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:30.945311 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:31.945824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:31.945354 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:31.945824 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:31.945483 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:32.945140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:32.945104 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:32.945140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:32.945147 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:32.945371 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:32.945255 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:32.945371 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:32.945356 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:33.945761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:33.945731 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:33.946208 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:33.945880 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:34.945552 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:34.945522 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:34.945713 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:34.945524 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:34.945713 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:34.945671 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:34.945818 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:34.945725 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:35.944957 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:35.944925 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:35.945145 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:35.945057 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:36.945561 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:36.945529 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:36.946049 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:36.945539 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:36.946049 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:36.945644 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:36.946049 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:36.945743 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:37.477986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:37.477953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:37.478130 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:37.477996 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:37.478130 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.478081 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:37.478130 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.478124 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:10:53.47811132 +0000 UTC m=+34.194488927 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:37.478228 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.478082 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:37.478228 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.478201 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:53.478186969 +0000 UTC m=+34.194564573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:37.678975 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:37.678937 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:37.679154 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.679097 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:37.679154 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.679117 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:37.679154 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.679125 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:37.679315 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.679172 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:10:53.679159505 +0000 UTC m=+34.395537108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:37.945756 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:37.945721 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:37.946182 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:37.945910 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:38.945346 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:38.945312 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:38.945494 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:38.945456 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:38.945568 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:38.945512 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:38.945662 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:38.945635 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:39.946209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:39.946175 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:39.946659 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:39.946285 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:40.945341 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:40.945312 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:40.945528 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:40.945351 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:40.945528 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:40.945435 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:40.945638 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:40.945569 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:41.946398 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:41.946154 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:41.947785 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:41.947741 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:42.054798 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.054766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" event={"ID":"b3eb08a4-b6f2-4d58-a93c-c0b962e5d365","Type":"ContainerStarted","Data":"b9ec16dec52440f3f2f2080f132d9f6be5390729cbbf1e178f5383e6e0f9e885"} Jul 10 13:10:42.056189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.056165 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-r4gbb" event={"ID":"615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde","Type":"ContainerStarted","Data":"0a1fd81f197577c315bf840f2c0b13cab23f990808578585d7f5c253a226859b"} Jul 10 13:10:42.057530 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.057506 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dc9tb" event={"ID":"00558e71-8d16-44ce-aa14-d69cdf75fb72","Type":"ContainerStarted","Data":"758b518d41ef0f24f0be24c28859d59a957d1e16986c8109c50d64885393a681"} Jul 10 13:10:42.059060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.059041 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" event={"ID":"13881716-db9c-451b-a633-763d9bbfc460","Type":"ContainerStarted","Data":"0dfeb76e198b2eb37e5141611b4f08ee93994c0e7d35b3825e6630e56cc3974d"} Jul 10 13:10:42.060503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.060485 2567 generic.go:356] "Generic (PLEG): container finished" podID="d9700277-f8d3-4ca9-8da5-7f728bfbd9a5" containerID="7dd638a768ad2dee41d9461aada27fbc5ee8acd33e466f9cf59d2ffe41d126a0" exitCode=0 Jul 10 13:10:42.060572 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.060551 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerDied","Data":"7dd638a768ad2dee41d9461aada27fbc5ee8acd33e466f9cf59d2ffe41d126a0"} Jul 10 13:10:42.062298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.062276 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hldt7" event={"ID":"30953d48-7c6d-4251-a21d-08c41361d4ba","Type":"ContainerStarted","Data":"8d8090998d83b309e1078ba08097b812015b5780ec93adc5dbf95cc7a195c6fc"} Jul 10 13:10:42.063856 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.063819 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-65xfz" event={"ID":"f6a6b054-52ca-4dfe-abf5-2ef50b8a3226","Type":"ContainerStarted","Data":"a1a3699ef8c23839078afecca4eae39481948e2c17c3dae90503a43e426e493f"} Jul 10 13:10:42.066046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.066030 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:10:42.066363 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.066346 2567 generic.go:356] "Generic (PLEG): container finished" podID="09abb34b-0b90-4794-9ff3-87a3f23fed12" containerID="7c4a0b53798fcd24536fdca7088a03dd5c5522208127f7aafc86a4609a9b71f0" exitCode=1 Jul 10 13:10:42.066429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.066371 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"bd4f580de006d08034531be9d3eabf4433d9f3c786bc7f0813d6f2b3f762aaf0"} Jul 10 13:10:42.066429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.066416 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"a8ad4dec6c10cb05c348115b6ff162c15140dc71f61a891253d0637a26cf05df"} Jul 10 13:10:42.066494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.066444 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerDied","Data":"7c4a0b53798fcd24536fdca7088a03dd5c5522208127f7aafc86a4609a9b71f0"} Jul 10 13:10:42.066494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.066473 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"c6625dcc198a3f22d0345b7c56f967eff371c8d1849aa937c9e47d89c4df1b82"} Jul 10 13:10:42.071262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.071221 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5gmgc" podStartSLOduration=2.538390904 podStartE2EDuration="22.071210698s" totalImagesPullingTime="19.532819794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:42.071047275 +0000 UTC m=+22.787424902" watchObservedRunningTime="2026-07-10 13:10:42.071210698 +0000 UTC m=+22.787588324" Jul 10 13:10:42.071384 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.071365 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-11.ec2.internal" podStartSLOduration=21.071360764 podStartE2EDuration="21.071360764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:26.040844519 +0000 UTC m=+6.757222149" watchObservedRunningTime="2026-07-10 13:10:42.071360764 +0000 UTC m=+22.787738389" Jul 10 13:10:42.088754 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.088719 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hldt7" podStartSLOduration=2.512391245 podStartE2EDuration="22.088708247s" totalImagesPullingTime="19.576317002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:42.088487295 +0000 UTC m=+22.804864920" watchObservedRunningTime="2026-07-10 13:10:42.088708247 +0000 UTC m=+22.805085873" Jul 10 13:10:42.104750 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.104696 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-65xfz" podStartSLOduration=2.508955418 podStartE2EDuration="22.104679537s" totalImagesPullingTime="19.595724119s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:42.104016213 +0000 UTC m=+22.820393839" watchObservedRunningTime="2026-07-10 13:10:42.104679537 +0000 UTC m=+22.821057165" Jul 10 13:10:42.119930 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.119891 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dc9tb" podStartSLOduration=10.566995701 podStartE2EDuration="22.119879581s" totalImagesPullingTime="11.55288388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:42.11952995 +0000 UTC m=+22.835907575" watchObservedRunningTime="2026-07-10 13:10:42.119879581 +0000 UTC m=+22.836257206" Jul 10 13:10:42.136752 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.136676 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-r4gbb" podStartSLOduration=2.596192432 podStartE2EDuration="22.136665094s" totalImagesPullingTime="19.540472662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:42.136470018 +0000 UTC m=+22.852847651" watchObservedRunningTime="2026-07-10 13:10:42.136665094 +0000 UTC m=+22.853042711" Jul 10 13:10:42.265549 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.265517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:42.368486 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.368453 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:42.945501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.945476 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:42.945693 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:42.945573 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:42.945693 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:42.945627 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:42.945768 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:42.945733 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:43.070470 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.070437 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8l2wn" event={"ID":"3494da59-74b5-4f5d-83dd-b9ae69096f05","Type":"ContainerStarted","Data":"c9b3de8f52c4c6af6cbd9e2d3a9f4ac461d9da10692e949b3530aae8c7bd91a5"} Jul 10 13:10:43.073771 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.073747 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:10:43.074227 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.074202 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"656d734793a6045a241e14e9b39100df8270695e1ae3e3ea1a39a81be71303d3"} Jul 10 13:10:43.074320 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.074281 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"f8cce8f24125ea12d55a5e93586a48f54eee8de76722e5bfb384d09a32349c5b"} Jul 10 13:10:43.085415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.085376 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8l2wn" podStartSLOduration=3.514439475 podStartE2EDuration="23.085361343s" totalImagesPullingTime="19.570921868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:43.085280266 +0000 UTC m=+23.801657893" watchObservedRunningTime="2026-07-10 13:10:43.085361343 +0000 UTC m=+23.801738969" Jul 10 13:10:43.202191 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.202128 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:43.265728 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.265706 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:43.479827 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.479803 2567 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 13:10:43.830405 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.830271 2567 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T13:10:43.479823352Z","UUID":"f8cc3de7-8865-4291-927b-76ca487b5d02","Handler":null,"Name":"","Endpoint":""} Jul 10 13:10:43.831658 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.831643 2567 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 13:10:43.831761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.831677 2567 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 13:10:43.945086 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:43.945044 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:43.945256 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:43.945182 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:44.077895 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.077863 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" event={"ID":"13881716-db9c-451b-a633-763d9bbfc460","Type":"ContainerStarted","Data":"e8ee1382399647a42df10204b9dd368f25430768da4dba88eaf1e7beef4d65c1"} Jul 10 13:10:44.460318 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.460286 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:44.561388 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.561312 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:44.762643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.762594 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:44.861181 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.861114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:44.945699 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.945677 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:44.945699 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:44.945696 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:44.945866 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:44.945798 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:44.945970 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:44.945909 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:45.081772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.081736 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" event={"ID":"13881716-db9c-451b-a633-763d9bbfc460","Type":"ContainerStarted","Data":"f52752dcca804fda92f8d123f0a29bbf6bc9289e35bd7222abd32cf11297cb25"} Jul 10 13:10:45.085224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.085200 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:10:45.085605 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.085584 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"0384299f8efc8b760dfcd4183a1701cbe5e9a79d0f72385abab2bcced46ec065"} Jul 10 13:10:45.103975 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.101185 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-99wz5" podStartSLOduration=2.577935742 podStartE2EDuration="25.101166617s" totalImagesPullingTime="22.523230875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:45.100713173 +0000 UTC m=+25.817090818" watchObservedRunningTime="2026-07-10 13:10:45.101166617 +0000 UTC m=+25.817544244" Jul 10 13:10:45.379161 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.378923 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:45.473713 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.473689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:45.759796 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.759763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:45.856322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.856291 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:45.945391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:45.945363 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:45.945524 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:45.945481 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:46.041462 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.041400 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:46.041957 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.041942 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:46.087877 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.087839 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:46.088294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.088278 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dc9tb" Jul 10 13:10:46.105425 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.105403 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:46.204705 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.204674 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:46.945718 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.945690 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:46.945888 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:46.945811 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:46.945888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:46.945869 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:46.946040 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:46.946001 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:47.094576 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:47.094542 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:10:47.095431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:47.095017 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"92e8ec4264f7f66475eb224b2d2539dde50db2c729b5f3e07e7aa283a1cd51ac"} Jul 10 13:10:47.095518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:47.095495 2567 scope.go:117] "RemoveContainer" containerID="7c4a0b53798fcd24536fdca7088a03dd5c5522208127f7aafc86a4609a9b71f0" Jul 10 13:10:47.945405 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:47.945377 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:47.945625 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:47.945521 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:48.758646 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:48.758621 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhc8c"] Jul 10 13:10:48.759062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:48.758750 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:48.759062 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:48.758875 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:48.761589 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:48.761567 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tmb52"] Jul 10 13:10:48.761675 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:48.761658 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:48.761734 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:48.761719 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:48.773156 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:48.773129 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dlr7k"] Jul 10 13:10:48.773262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:48.773253 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:48.773379 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:48.773357 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:49.103216 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.103163 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:10:49.103590 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.103560 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" event={"ID":"09abb34b-0b90-4794-9ff3-87a3f23fed12","Type":"ContainerStarted","Data":"be3c4be6b0b50999ea82a41a5cc6e5f3b5330304d90b7e90f688e00eeeae4ee9"} Jul 10 13:10:49.103708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.103602 2567 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 13:10:49.103806 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.103782 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:49.103913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.103866 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:49.105595 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.105569 2567 generic.go:356] "Generic (PLEG): container finished" podID="d9700277-f8d3-4ca9-8da5-7f728bfbd9a5" containerID="5584f2c939cd80e42815cd9cef3f78e12cc3d133ab314f418fff5b900ca5d881" exitCode=0 Jul 10 13:10:49.105682 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.105608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerDied","Data":"5584f2c939cd80e42815cd9cef3f78e12cc3d133ab314f418fff5b900ca5d881"} Jul 10 13:10:49.119457 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.119440 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:49.119534 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.119529 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:49.132014 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.131973 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" podStartSLOduration=9.436748969 podStartE2EDuration="29.131961037s" totalImagesPullingTime="19.695212068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:10:49.130426969 +0000 UTC m=+29.846804595" watchObservedRunningTime="2026-07-10 13:10:49.131961037 +0000 UTC m=+29.848338641" Jul 10 13:10:49.642930 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.642905 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:10:49.946935 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.946690 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:49.947352 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.946772 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:49.947352 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:49.946995 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:49.947352 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:49.947130 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:49.947352 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:49.946811 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:49.947352 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:49.947255 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:50.439625 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:50.439598 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:50.540717 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:50.540673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:51.113135 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.113111 2567 generic.go:356] "Generic (PLEG): container finished" podID="d9700277-f8d3-4ca9-8da5-7f728bfbd9a5" containerID="0964edf15a023325abca6a92ee7f0d27f2e74639c838aef283eee13215e10183" exitCode=0 Jul 10 13:10:51.113555 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.113193 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerDied","Data":"0964edf15a023325abca6a92ee7f0d27f2e74639c838aef283eee13215e10183"} Jul 10 13:10:51.123057 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.123030 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" podUID="09abb34b-0b90-4794-9ff3-87a3f23fed12" containerName="ovnkube-controller" probeResult="failure" output="" Jul 10 13:10:51.387693 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.387620 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:51.453071 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.453040 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:51.804358 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.804323 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:51.904134 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.904108 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:51.945386 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.945363 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:51.945498 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.945363 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:51.945498 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:51.945485 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:51.945598 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:51.945506 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:51.945965 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:51.945594 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:51.948733 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:51.946067 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:52.004599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:52.004577 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:52.105508 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:52.105451 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:52.116983 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:52.116957 2567 generic.go:356] "Generic (PLEG): container finished" podID="d9700277-f8d3-4ca9-8da5-7f728bfbd9a5" containerID="36ebf106afde295c92b3884f8db3c00241d936ada99f60dd3413f5c65b0a826c" exitCode=0 Jul 10 13:10:52.117373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:52.116991 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerDied","Data":"36ebf106afde295c92b3884f8db3c00241d936ada99f60dd3413f5c65b0a826c"} Jul 10 13:10:52.205403 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:52.205380 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:52.566439 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:52.566413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r4gbb_615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde/dns-node-resolver/0.log" Jul 10 13:10:53.495398 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.495372 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:53.495902 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.495409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:53.495902 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.495501 2567 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:53.495902 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.495503 2567 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:53.495902 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.495546 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs podName:da325937-f2c1-4ffe-ade9-580cd62f69ef nodeName:}" failed. No retries permitted until 2026-07-10 13:11:25.495532833 +0000 UTC m=+66.211910437 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs") pod "network-metrics-daemon-dlr7k" (UID: "da325937-f2c1-4ffe-ade9-580cd62f69ef") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 13:10:53.495902 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.495558 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret podName:a311c9cc-4ec0-4099-a20c-e85c5e1c6801 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:25.495552238 +0000 UTC m=+66.211929841 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret") pod "global-pull-secret-syncer-xhc8c" (UID: "a311c9cc-4ec0-4099-a20c-e85c5e1c6801") : object "kube-system"/"original-pull-secret" not registered Jul 10 13:10:53.566102 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.566078 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65xfz_f6a6b054-52ca-4dfe-abf5-2ef50b8a3226/node-ca/0.log" Jul 10 13:10:53.697490 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.697460 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:53.697683 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.697631 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 13:10:53.697683 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.697652 2567 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 13:10:53.697683 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.697663 2567 projected.go:196] Error preparing data for projected volume kube-api-access-htf7w for pod openshift-network-diagnostics/network-check-target-tmb52: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:53.697868 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.697724 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w podName:25da3fbb-7215-4da2-af7a-ab6148889586 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:25.6977061 +0000 UTC m=+66.414083703 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-htf7w" (UniqueName: "kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w") pod "network-check-target-tmb52" (UID: "25da3fbb-7215-4da2-af7a-ab6148889586") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 13:10:53.699248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.699218 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:53.753302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.753238 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:53.945744 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.945711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:53.945915 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.945711 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:53.945915 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.945823 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tmb52" podUID="25da3fbb-7215-4da2-af7a-ab6148889586" Jul 10 13:10:53.945915 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:53.945715 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:53.946085 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.945946 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlr7k" podUID="da325937-f2c1-4ffe-ade9-580cd62f69ef" Jul 10 13:10:53.946085 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:10:53.945998 2567 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhc8c" podUID="a311c9cc-4ec0-4099-a20c-e85c5e1c6801" Jul 10 13:10:54.603497 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:54.603467 2567 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-11.ec2.internal" event="NodeReady" Jul 10 13:10:54.604026 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:54.603592 2567 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 13:10:54.836091 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:54.836060 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:55.402768 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.402741 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:55.721077 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.721033 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:55.821412 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.821379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:55.925274 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.925238 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:55.945197 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.945172 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:10:55.945347 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.945201 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:10:55.945347 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.945176 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:10:55.949327 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.949306 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnrp9\"" Jul 10 13:10:55.949448 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.949340 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 13:10:55.949448 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.949363 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xcvz\"" Jul 10 13:10:55.949448 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.949375 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 13:10:55.949612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.949508 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 13:10:55.949668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:55.949610 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 13:10:56.037483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:56.037418 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:56.105359 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:56.105332 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:57.105257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:57.105215 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:57.486263 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:57.486116 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:57.586577 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:57.586556 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:57.702130 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:57.702100 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:57.911191 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:57.911162 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:58.099342 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:58.099311 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:58.132191 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:58.132168 2567 generic.go:356] "Generic (PLEG): container finished" podID="d9700277-f8d3-4ca9-8da5-7f728bfbd9a5" containerID="b3d7f7399f6489b3b087fcee3de10d07c914acf0721deb58d7c81e95fe74660a" exitCode=0 Jul 10 13:10:58.132456 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:58.132215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerDied","Data":"b3d7f7399f6489b3b087fcee3de10d07c914acf0721deb58d7c81e95fe74660a"} Jul 10 13:10:58.167373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:58.167346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.001246 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.001221 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.117313 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.117273 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.136772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.136744 2567 generic.go:356] "Generic (PLEG): container finished" podID="d9700277-f8d3-4ca9-8da5-7f728bfbd9a5" containerID="4d6b5d4728e3227ba6bb8412d63edd9988a6cc908d532222fd8e7d4b932f86ba" exitCode=0 Jul 10 13:10:59.137094 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.136806 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerDied","Data":"4d6b5d4728e3227ba6bb8412d63edd9988a6cc908d532222fd8e7d4b932f86ba"} Jul 10 13:10:59.191469 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.191441 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.303100 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.303065 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.703816 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.703789 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.803427 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.803404 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:10:59.905369 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:10:59.905344 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:00.141897 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:00.141821 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98dtr" event={"ID":"d9700277-f8d3-4ca9-8da5-7f728bfbd9a5","Type":"ContainerStarted","Data":"16648e0f59a67faab704f1f9da99f8a6667e6c83b32898e9bd1f55ca8f4c5bc5"} Jul 10 13:11:00.189261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:00.189218 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-98dtr" podStartSLOduration=6.751426829 podStartE2EDuration="40.189204484s" totalImagesPullingTime="33.437777655s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:00.188941296 +0000 UTC m=+40.905318923" watchObservedRunningTime="2026-07-10 13:11:00.189204484 +0000 UTC m=+40.905582111" Jul 10 13:11:00.311443 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:00.311417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:00.412172 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:00.412115 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.100419 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.100388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.207308 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.207283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.301787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.301763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.362501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.362442 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.499032 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.499006 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.551658 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.551637 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.737010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.736988 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.798736 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.798714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.917995 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.917970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:01.979290 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:01.979266 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.147523 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.147470 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.203899 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.203872 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.270104 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.270065 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.370619 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.370591 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.470232 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.470207 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.531881 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.531858 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:02.914189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:02.914125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:03.102134 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:03.102106 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:03.158584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:03.158559 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:03.353182 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:03.353155 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:03.505599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:03.505568 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:04.097801 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:04.097749 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:04.178501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:04.178476 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:04.507209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:04.507184 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:04.607830 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:04.607806 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:04.928277 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:04.928205 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:05.109716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:05.109687 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:05.495367 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:05.495336 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:05.764325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:05.764259 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:06.105117 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:06.105053 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:06.301780 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:06.301751 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:06.496421 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:06.496394 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:06.916552 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:06.916493 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:06.980483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:06.980464 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:12.586133 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:12.586085 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:15.954071 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:15.954032 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:17.787371 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.787335 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qdk2c"] Jul 10 13:11:17.789725 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.789709 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hnqr4"] Jul 10 13:11:17.789869 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.789836 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.791875 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.791819 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.792175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.792158 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vrvn5\"" Jul 10 13:11:17.792292 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.792271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:17.793055 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.793036 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 13:11:17.793182 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.793054 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 13:11:17.795914 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.795895 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 13:11:17.796014 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.795979 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ndhrp\"" Jul 10 13:11:17.796971 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.796934 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6fd5697954-ngqvd"] Jul 10 13:11:17.799656 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.799640 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.800876 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.800839 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 13:11:17.800996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.800977 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 13:11:17.804481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.804462 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 13:11:17.804553 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.804509 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-ngc6m\"" Jul 10 13:11:17.804653 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.804639 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 13:11:17.804775 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.804760 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 13:11:17.811039 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.811019 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hnqr4"] Jul 10 13:11:17.814464 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.814444 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdk2c"] Jul 10 13:11:17.815222 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.815201 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 13:11:17.820617 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.820594 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd5697954-ngqvd"] Jul 10 13:11:17.840405 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.840383 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kw8td"] Jul 10 13:11:17.842573 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.842557 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.845227 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.845212 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 13:11:17.845444 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.845426 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 13:11:17.845694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.845678 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 13:11:17.845885 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.845872 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 13:11:17.845963 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.845925 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4zgls\"" Jul 10 13:11:17.847712 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847695 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a787e51b-809a-4735-9f40-2d3331f1824d-config-volume\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.847792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a787e51b-809a-4735-9f40-2d3331f1824d-metrics-tls\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.847792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847744 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a787e51b-809a-4735-9f40-2d3331f1824d-tmp-dir\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.847914 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847803 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aed2557-b2e1-499b-807e-4d61ad1f81ef-registry-certificates\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.847914 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847833 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aed2557-b2e1-499b-807e-4d61ad1f81ef-installation-pull-secrets\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.847914 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp5q\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-kube-api-access-sfp5q\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.848052 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847921 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-crio-socket\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.848052 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847954 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.848052 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.847983 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.848052 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848013 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkznm\" (UniqueName: \"kubernetes.io/projected/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-kube-api-access-nkznm\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.848052 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848045 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aed2557-b2e1-499b-807e-4d61ad1f81ef-ca-trust-extracted\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.848267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7bss\" (UniqueName: \"kubernetes.io/projected/50f2aa9c-9afd-42a4-9155-2d2006ba5b50-kube-api-access-q7bss\") pod \"ingress-canary-hnqr4\" (UID: \"50f2aa9c-9afd-42a4-9155-2d2006ba5b50\") " pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.848267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-data-volume\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.848267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848168 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjb74\" (UniqueName: \"kubernetes.io/projected/a787e51b-809a-4735-9f40-2d3331f1824d-kube-api-access-fjb74\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.848267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aed2557-b2e1-499b-807e-4d61ad1f81ef-image-registry-private-configuration\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.848557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848291 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f2aa9c-9afd-42a4-9155-2d2006ba5b50-cert\") pod \"ingress-canary-hnqr4\" (UID: \"50f2aa9c-9afd-42a4-9155-2d2006ba5b50\") " pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.848557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aed2557-b2e1-499b-807e-4d61ad1f81ef-trusted-ca\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.848557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848378 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-bound-sa-token\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.848557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.848453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-registry-tls\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.858641 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.858617 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kw8td"] Jul 10 13:11:17.948714 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948691 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a787e51b-809a-4735-9f40-2d3331f1824d-config-volume\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.948864 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a787e51b-809a-4735-9f40-2d3331f1824d-metrics-tls\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.948864 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948743 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a787e51b-809a-4735-9f40-2d3331f1824d-tmp-dir\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.948864 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aed2557-b2e1-499b-807e-4d61ad1f81ef-registry-certificates\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.948864 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948789 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aed2557-b2e1-499b-807e-4d61ad1f81ef-installation-pull-secrets\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949066 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948910 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp5q\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-kube-api-access-sfp5q\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949066 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948952 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-crio-socket\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.949066 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.948982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.949066 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.949066 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkznm\" (UniqueName: \"kubernetes.io/projected/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-kube-api-access-nkznm\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.949066 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aed2557-b2e1-499b-807e-4d61ad1f81ef-ca-trust-extracted\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949079 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7bss\" (UniqueName: \"kubernetes.io/projected/50f2aa9c-9afd-42a4-9155-2d2006ba5b50-kube-api-access-q7bss\") pod \"ingress-canary-hnqr4\" (UID: \"50f2aa9c-9afd-42a4-9155-2d2006ba5b50\") " pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949108 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-data-volume\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949142 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjb74\" (UniqueName: \"kubernetes.io/projected/a787e51b-809a-4735-9f40-2d3331f1824d-kube-api-access-fjb74\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949162 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a787e51b-809a-4735-9f40-2d3331f1824d-tmp-dir\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949183 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aed2557-b2e1-499b-807e-4d61ad1f81ef-image-registry-private-configuration\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949210 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f2aa9c-9afd-42a4-9155-2d2006ba5b50-cert\") pod \"ingress-canary-hnqr4\" (UID: \"50f2aa9c-9afd-42a4-9155-2d2006ba5b50\") " pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949245 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aed2557-b2e1-499b-807e-4d61ad1f81ef-trusted-ca\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949268 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-bound-sa-token\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949355 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949304 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-registry-tls\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949389 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a787e51b-809a-4735-9f40-2d3331f1824d-config-volume\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.949787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949461 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-crio-socket\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.949787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949639 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8aed2557-b2e1-499b-807e-4d61ad1f81ef-registry-certificates\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.949787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949702 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.950008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949947 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-data-volume\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.950008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.949960 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8aed2557-b2e1-499b-807e-4d61ad1f81ef-ca-trust-extracted\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.950913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.950888 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8aed2557-b2e1-499b-807e-4d61ad1f81ef-trusted-ca\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.953247 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.953221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a787e51b-809a-4735-9f40-2d3331f1824d-metrics-tls\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.953342 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.953303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-registry-tls\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.953400 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.953368 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8aed2557-b2e1-499b-807e-4d61ad1f81ef-image-registry-private-configuration\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.953436 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.953401 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.953436 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.953409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8aed2557-b2e1-499b-807e-4d61ad1f81ef-installation-pull-secrets\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.953510 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.953487 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50f2aa9c-9afd-42a4-9155-2d2006ba5b50-cert\") pod \"ingress-canary-hnqr4\" (UID: \"50f2aa9c-9afd-42a4-9155-2d2006ba5b50\") " pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.962150 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.962119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjb74\" (UniqueName: \"kubernetes.io/projected/a787e51b-809a-4735-9f40-2d3331f1824d-kube-api-access-fjb74\") pod \"dns-default-qdk2c\" (UID: \"a787e51b-809a-4735-9f40-2d3331f1824d\") " pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:17.962623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.962600 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkznm\" (UniqueName: \"kubernetes.io/projected/29e1b4e5-a9ec-44f7-b0a4-e40181929e18-kube-api-access-nkznm\") pod \"insights-runtime-extractor-kw8td\" (UID: \"29e1b4e5-a9ec-44f7-b0a4-e40181929e18\") " pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:17.963921 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.963903 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp5q\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-kube-api-access-sfp5q\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.964117 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.964096 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8aed2557-b2e1-499b-807e-4d61ad1f81ef-bound-sa-token\") pod \"image-registry-6fd5697954-ngqvd\" (UID: \"8aed2557-b2e1-499b-807e-4d61ad1f81ef\") " pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:17.964427 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.964408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7bss\" (UniqueName: \"kubernetes.io/projected/50f2aa9c-9afd-42a4-9155-2d2006ba5b50-kube-api-access-q7bss\") pod \"ingress-canary-hnqr4\" (UID: \"50f2aa9c-9afd-42a4-9155-2d2006ba5b50\") " pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:17.988420 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:17.988398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:18.092999 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.092939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:18.099905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.099881 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:18.105289 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.105266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hnqr4" Jul 10 13:11:18.110993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.110970 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:18.150570 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.150469 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kw8td" Jul 10 13:11:18.275183 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.275157 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdk2c"] Jul 10 13:11:18.277868 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:18.277827 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda787e51b_809a_4735_9f40_2d3331f1824d.slice/crio-d91308d4961a6ac8d6cbf6be17255b7aab014b633961f7744def1f09d3f9812d WatchSource:0}: Error finding container d91308d4961a6ac8d6cbf6be17255b7aab014b633961f7744def1f09d3f9812d: Status 404 returned error can't find the container with id d91308d4961a6ac8d6cbf6be17255b7aab014b633961f7744def1f09d3f9812d Jul 10 13:11:18.318501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.318478 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:18.318501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.318495 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kw8td"] Jul 10 13:11:18.319632 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:18.319608 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e1b4e5_a9ec_44f7_b0a4_e40181929e18.slice/crio-480c6c45569f7ee1b0649c36636b44f306d9f5463fb0b125c38c066a0f3b7436 WatchSource:0}: Error finding container 480c6c45569f7ee1b0649c36636b44f306d9f5463fb0b125c38c066a0f3b7436: Status 404 returned error can't find the container with id 480c6c45569f7ee1b0649c36636b44f306d9f5463fb0b125c38c066a0f3b7436 Jul 10 13:11:18.414321 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.414248 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:18.524658 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.524623 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6fd5697954-ngqvd"] Jul 10 13:11:18.525226 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.525204 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hnqr4"] Jul 10 13:11:18.525339 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:18.525272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f2aa9c_9afd_42a4_9155_2d2006ba5b50.slice/crio-1cc21f07faf052ca2a6a69453e687b5104601705f7ea3e2bd0f42c4a58a5850f WatchSource:0}: Error finding container 1cc21f07faf052ca2a6a69453e687b5104601705f7ea3e2bd0f42c4a58a5850f: Status 404 returned error can't find the container with id 1cc21f07faf052ca2a6a69453e687b5104601705f7ea3e2bd0f42c4a58a5850f Jul 10 13:11:18.525832 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:18.525813 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aed2557_b2e1_499b_807e_4d61ad1f81ef.slice/crio-44061b0108da3100f947be12a79011d44d3b7b3abe40cb3ed28f52aab88b0d89 WatchSource:0}: Error finding container 44061b0108da3100f947be12a79011d44d3b7b3abe40cb3ed28f52aab88b0d89: Status 404 returned error can't find the container with id 44061b0108da3100f947be12a79011d44d3b7b3abe40cb3ed28f52aab88b0d89 Jul 10 13:11:18.592897 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.592876 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:18.693392 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:18.693318 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:19.185974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:19.185940 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hnqr4" event={"ID":"50f2aa9c-9afd-42a4-9155-2d2006ba5b50","Type":"ContainerStarted","Data":"1cc21f07faf052ca2a6a69453e687b5104601705f7ea3e2bd0f42c4a58a5850f"} Jul 10 13:11:19.191333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:19.191300 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" event={"ID":"8aed2557-b2e1-499b-807e-4d61ad1f81ef","Type":"ContainerStarted","Data":"94f655f66b6cc7c6cf1ac0ab8a96c89dbd74f7faab2e18c60c8f6c4fdd2fa6a9"} Jul 10 13:11:19.191466 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:19.191353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" event={"ID":"8aed2557-b2e1-499b-807e-4d61ad1f81ef","Type":"ContainerStarted","Data":"44061b0108da3100f947be12a79011d44d3b7b3abe40cb3ed28f52aab88b0d89"} Jul 10 13:11:19.192092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:19.192048 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:19.194008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:19.193935 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdk2c" event={"ID":"a787e51b-809a-4735-9f40-2d3331f1824d","Type":"ContainerStarted","Data":"d91308d4961a6ac8d6cbf6be17255b7aab014b633961f7744def1f09d3f9812d"} Jul 10 13:11:19.198694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:19.198665 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8td" event={"ID":"29e1b4e5-a9ec-44f7-b0a4-e40181929e18","Type":"ContainerStarted","Data":"480c6c45569f7ee1b0649c36636b44f306d9f5463fb0b125c38c066a0f3b7436"} Jul 10 13:11:20.551788 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.551753 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:20.637459 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.637416 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:20.727716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.727670 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" podStartSLOduration=3.727653666 podStartE2EDuration="3.727653666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:19.238142121 +0000 UTC m=+59.954519748" watchObservedRunningTime="2026-07-10 13:11:20.727653666 +0000 UTC m=+61.444031285" Jul 10 13:11:20.728412 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.728391 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht"] Jul 10 13:11:20.729335 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.729304 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:20.737674 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.737656 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:20.740453 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.740362 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 13:11:20.741023 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.741004 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bm7zj\"" Jul 10 13:11:20.741313 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.741295 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht"] Jul 10 13:11:20.784892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.784871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9afcc22-2cac-4501-ac9f-c15fd97c03d6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-x9kht\" (UID: \"c9afcc22-2cac-4501-ac9f-c15fd97c03d6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:20.809505 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.809450 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:20.879908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.879872 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:20.885950 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.885923 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9afcc22-2cac-4501-ac9f-c15fd97c03d6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-x9kht\" (UID: \"c9afcc22-2cac-4501-ac9f-c15fd97c03d6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:20.888429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:20.888409 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c9afcc22-2cac-4501-ac9f-c15fd97c03d6-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-x9kht\" (UID: \"c9afcc22-2cac-4501-ac9f-c15fd97c03d6\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:21.046787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:21.046755 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:21.123950 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:21.123922 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knf4x" Jul 10 13:11:21.294524 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:21.294499 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:21.396730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:21.396649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:21.772727 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:21.772693 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:21.939688 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:21.939658 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:22.816205 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:22.816164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:22.903032 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:22.903004 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:23.793943 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:23.793825 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht"] Jul 10 13:11:23.797701 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:23.797663 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9afcc22_2cac_4501_ac9f_c15fd97c03d6.slice/crio-60619bf6fb011db208049185e2c6800390d001d31843ce36318f970d365fb0cd WatchSource:0}: Error finding container 60619bf6fb011db208049185e2c6800390d001d31843ce36318f970d365fb0cd: Status 404 returned error can't find the container with id 60619bf6fb011db208049185e2c6800390d001d31843ce36318f970d365fb0cd Jul 10 13:11:24.212690 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.212653 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hnqr4" event={"ID":"50f2aa9c-9afd-42a4-9155-2d2006ba5b50","Type":"ContainerStarted","Data":"5642bd3b7000efcf9d2971364fa30a96c37ee7c3777c61e03d502bb3e2ae6fcd"} Jul 10 13:11:24.213775 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.213751 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" event={"ID":"c9afcc22-2cac-4501-ac9f-c15fd97c03d6","Type":"ContainerStarted","Data":"60619bf6fb011db208049185e2c6800390d001d31843ce36318f970d365fb0cd"} Jul 10 13:11:24.215420 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.215402 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdk2c" event={"ID":"a787e51b-809a-4735-9f40-2d3331f1824d","Type":"ContainerStarted","Data":"aa4af3af1f8defb7001bec12c6eb4502f38d0e055c7a31ddd74e1977cf695ed0"} Jul 10 13:11:24.215497 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.215438 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdk2c" event={"ID":"a787e51b-809a-4735-9f40-2d3331f1824d","Type":"ContainerStarted","Data":"a44c582c9d05b7d56b2b14d78e8ca29087ae9962f40d317ce55ee16e953f6123"} Jul 10 13:11:24.215497 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.215459 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:24.216824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.216805 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8td" event={"ID":"29e1b4e5-a9ec-44f7-b0a4-e40181929e18","Type":"ContainerStarted","Data":"08b9bea7bd43b3fb8ab481a6cbf90beb56cffd190c2d983441e6e8dbfc69f3a9"} Jul 10 13:11:24.231750 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.231708 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hnqr4" podStartSLOduration=2.101569939 podStartE2EDuration="7.231695032s" totalImagesPullingTime="5.130125093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:24.231378577 +0000 UTC m=+64.947756221" watchObservedRunningTime="2026-07-10 13:11:24.231695032 +0000 UTC m=+64.948072657" Jul 10 13:11:24.260385 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:24.260342 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qdk2c" podStartSLOduration=1.885416292 podStartE2EDuration="7.260330703s" totalImagesPullingTime="5.374914411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:24.259335842 +0000 UTC m=+64.975713468" watchObservedRunningTime="2026-07-10 13:11:24.260330703 +0000 UTC m=+64.976708326" Jul 10 13:11:25.223428 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.223026 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" event={"ID":"c9afcc22-2cac-4501-ac9f-c15fd97c03d6","Type":"ContainerStarted","Data":"c805dc7978e394a751b6268976d5edbf29f22f074826f2861bb55b99f9ffd8d8"} Jul 10 13:11:25.223428 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.223206 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:25.224804 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.224778 2567 patch_prober.go:28] interesting pod/prometheus-operator-admission-webhook-77d98bc786-x9kht container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.133.0.10:8443/healthz\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body= Jul 10 13:11:25.224923 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.224826 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" podUID="c9afcc22-2cac-4501-ac9f-c15fd97c03d6" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.133.0.10:8443/healthz\": dial tcp 10.133.0.10:8443: connect: connection refused" Jul 10 13:11:25.225459 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.225348 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8td" event={"ID":"29e1b4e5-a9ec-44f7-b0a4-e40181929e18","Type":"ContainerStarted","Data":"b3871909b481542ab0302026e2e1c3c300091ec37a9dfff0b37779e344289835"} Jul 10 13:11:25.240437 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.240392 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" podStartSLOduration=3.908367691 podStartE2EDuration="5.240374816s" totalImagesPullingTime="1.332007125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:25.239142734 +0000 UTC m=+65.955520366" watchObservedRunningTime="2026-07-10 13:11:25.240374816 +0000 UTC m=+65.956752435" Jul 10 13:11:25.523372 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.523341 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:11:25.523526 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.523397 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:11:25.526795 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.526773 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 13:11:25.528156 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.528140 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 13:11:25.536774 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.536748 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da325937-f2c1-4ffe-ade9-580cd62f69ef-metrics-certs\") pod \"network-metrics-daemon-dlr7k\" (UID: \"da325937-f2c1-4ffe-ade9-580cd62f69ef\") " pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:11:25.536774 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.536766 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a311c9cc-4ec0-4099-a20c-e85c5e1c6801-original-pull-secret\") pod \"global-pull-secret-syncer-xhc8c\" (UID: \"a311c9cc-4ec0-4099-a20c-e85c5e1c6801\") " pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:11:25.664301 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.664266 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhc8c" Jul 10 13:11:25.672659 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.672512 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xcvz\"" Jul 10 13:11:25.679986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.679796 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlr7k" Jul 10 13:11:25.725576 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.725529 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:11:25.728431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.728288 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 13:11:25.741040 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.739465 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 13:11:25.750632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.750603 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htf7w\" (UniqueName: \"kubernetes.io/projected/25da3fbb-7215-4da2-af7a-ab6148889586-kube-api-access-htf7w\") pod \"network-check-target-tmb52\" (UID: \"25da3fbb-7215-4da2-af7a-ab6148889586\") " pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:11:25.816319 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.816284 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhc8c"] Jul 10 13:11:25.819302 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:25.819262 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda311c9cc_4ec0_4099_a20c_e85c5e1c6801.slice/crio-21f9bb9a681a3224403303101c086b46eb8c0fd9e7b9a6fad4d5f64fd71fc76c WatchSource:0}: Error finding container 21f9bb9a681a3224403303101c086b46eb8c0fd9e7b9a6fad4d5f64fd71fc76c: Status 404 returned error can't find the container with id 21f9bb9a681a3224403303101c086b46eb8c0fd9e7b9a6fad4d5f64fd71fc76c Jul 10 13:11:25.832825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.832796 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dlr7k"] Jul 10 13:11:25.836486 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:25.836460 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda325937_f2c1_4ffe_ade9_580cd62f69ef.slice/crio-60dbb2b1c70f641c78b59bb6504b6b2a19f196d83d05e1cf5e26ed3d4a20df88 WatchSource:0}: Error finding container 60dbb2b1c70f641c78b59bb6504b6b2a19f196d83d05e1cf5e26ed3d4a20df88: Status 404 returned error can't find the container with id 60dbb2b1c70f641c78b59bb6504b6b2a19f196d83d05e1cf5e26ed3d4a20df88 Jul 10 13:11:25.960204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.960181 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gnrp9\"" Jul 10 13:11:25.968049 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:25.968026 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:11:26.112425 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.112363 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tmb52"] Jul 10 13:11:26.115155 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:26.115128 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25da3fbb_7215_4da2_af7a_ab6148889586.slice/crio-7d57644171fcf03a6eae4f0860749a52bed89e384f81e9166e79000707058088 WatchSource:0}: Error finding container 7d57644171fcf03a6eae4f0860749a52bed89e384f81e9166e79000707058088: Status 404 returned error can't find the container with id 7d57644171fcf03a6eae4f0860749a52bed89e384f81e9166e79000707058088 Jul 10 13:11:26.231311 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.231274 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tmb52" event={"ID":"25da3fbb-7215-4da2-af7a-ab6148889586","Type":"ContainerStarted","Data":"7d57644171fcf03a6eae4f0860749a52bed89e384f81e9166e79000707058088"} Jul 10 13:11:26.232625 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.232595 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dlr7k" event={"ID":"da325937-f2c1-4ffe-ade9-580cd62f69ef","Type":"ContainerStarted","Data":"60dbb2b1c70f641c78b59bb6504b6b2a19f196d83d05e1cf5e26ed3d4a20df88"} Jul 10 13:11:26.233793 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.233766 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhc8c" event={"ID":"a311c9cc-4ec0-4099-a20c-e85c5e1c6801","Type":"ContainerStarted","Data":"21f9bb9a681a3224403303101c086b46eb8c0fd9e7b9a6fad4d5f64fd71fc76c"} Jul 10 13:11:26.238142 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.238120 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-x9kht" Jul 10 13:11:26.784135 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.784105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:26.899787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:26.899735 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:27.389199 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:27.389139 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:28.244497 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:28.244456 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kw8td" event={"ID":"29e1b4e5-a9ec-44f7-b0a4-e40181929e18","Type":"ContainerStarted","Data":"0a8cb3bf7d2d2cab2adefb86c45a414d768883704f72c3409cf0e29e599df31b"} Jul 10 13:11:28.265593 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:28.265538 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kw8td" podStartSLOduration=2.028500493 podStartE2EDuration="11.265517918s" totalImagesPullingTime="9.237017425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:11:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:28.264153315 +0000 UTC m=+68.980530957" watchObservedRunningTime="2026-07-10 13:11:28.265517918 +0000 UTC m=+68.981895547" Jul 10 13:11:28.893839 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:28.893759 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:28.994420 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:28.994391 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:29.251208 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:29.251168 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dlr7k" event={"ID":"da325937-f2c1-4ffe-ade9-580cd62f69ef","Type":"ContainerStarted","Data":"24671f86718007e5a085b36f6b3bf1ef4b0102495b509203b1837f7440ebee83"} Jul 10 13:11:29.251363 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:29.251231 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dlr7k" event={"ID":"da325937-f2c1-4ffe-ade9-580cd62f69ef","Type":"ContainerStarted","Data":"91b9889ccbe9b06b54c790a565561b0ef2428ceec8516cbfc882839db152a2fc"} Jul 10 13:11:29.269881 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:29.269810 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dlr7k" podStartSLOduration=66.719786351 podStartE2EDuration="1m9.269793863s" totalImagesPullingTime="2.550007512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:29.269589289 +0000 UTC m=+69.985966916" watchObservedRunningTime="2026-07-10 13:11:29.269793863 +0000 UTC m=+69.986171490" Jul 10 13:11:29.398821 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:29.398773 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:29.502520 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:29.502439 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:30.963799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:30.963763 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:31.060166 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:31.060127 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:31.163037 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:31.163011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:31.262634 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:31.262540 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:33.268044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:33.268000 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhc8c" event={"ID":"a311c9cc-4ec0-4099-a20c-e85c5e1c6801","Type":"ContainerStarted","Data":"710e7d634419a8a3cf85af28ba5d196436c900b720caa6e6f3dc245d30894d84"} Jul 10 13:11:33.269479 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:33.269458 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tmb52" event={"ID":"25da3fbb-7215-4da2-af7a-ab6148889586","Type":"ContainerStarted","Data":"ec06b82bd999e358cb01c4edb81d454f266c005e979d7bba2983d7bae19c3734"} Jul 10 13:11:33.269608 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:33.269591 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:11:33.302774 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:33.302730 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xhc8c" podStartSLOduration=66.738117789 podStartE2EDuration="1m13.302717698s" totalImagesPullingTime="6.564599909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:33.302152335 +0000 UTC m=+74.018529978" watchObservedRunningTime="2026-07-10 13:11:33.302717698 +0000 UTC m=+74.019095314" Jul 10 13:11:34.086878 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.086811 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:34.227461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.227438 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qdk2c" Jul 10 13:11:34.279758 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.279719 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tmb52" podStartSLOduration=68.014321327 podStartE2EDuration="1m14.279705309s" totalImagesPullingTime="6.265383982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:10:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:33.337005658 +0000 UTC m=+74.053383284" watchObservedRunningTime="2026-07-10 13:11:34.279705309 +0000 UTC m=+74.996082974" Jul 10 13:11:34.490277 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.490243 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:34.519349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.519326 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zd2f7"] Jul 10 13:11:34.521598 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.521583 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.535514 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.535497 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 10 13:11:34.535599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.535526 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 13:11:34.535599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.535501 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 13:11:34.535664 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.535508 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 10 13:11:34.535916 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.535901 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9txk8\"" Jul 10 13:11:34.535955 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.535944 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 13:11:34.549411 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.549397 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 10 13:11:34.581782 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.581763 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9d576ed-1c8b-4be1-91ad-7e672b29f503-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.581891 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.581818 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.581891 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.581838 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.581891 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.581879 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58ms8\" (UniqueName: \"kubernetes.io/projected/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-api-access-58ms8\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.582010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.581928 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f9d576ed-1c8b-4be1-91ad-7e672b29f503-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.582010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.581944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.634429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.634410 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zd2f7"] Jul 10 13:11:34.646769 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.646750 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:34.682739 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.682714 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.682832 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.682746 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.682832 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.682772 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58ms8\" (UniqueName: \"kubernetes.io/projected/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-api-access-58ms8\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.682934 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.682859 2567 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 10 13:11:34.682934 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.682906 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-tls podName:f9d576ed-1c8b-4be1-91ad-7e672b29f503 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:35.18289117 +0000 UTC m=+75.899268775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-zd2f7" (UID: "f9d576ed-1c8b-4be1-91ad-7e672b29f503") : secret "kube-state-metrics-tls" not found Jul 10 13:11:34.683012 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.682973 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f9d576ed-1c8b-4be1-91ad-7e672b29f503-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.683048 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.683015 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.683090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.683054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9d576ed-1c8b-4be1-91ad-7e672b29f503-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.683303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.683283 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f9d576ed-1c8b-4be1-91ad-7e672b29f503-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.683467 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.683451 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.683566 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.683550 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9d576ed-1c8b-4be1-91ad-7e672b29f503-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.685239 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.685220 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.719459 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.719431 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58ms8\" (UniqueName: \"kubernetes.io/projected/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-api-access-58ms8\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:34.750073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.750006 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n4shw"] Jul 10 13:11:34.753041 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.753020 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.757305 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.757277 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-t6hvq\" is forbidden: User \"system:node:ip-10-0-140-11.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-11.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6hvq\"" type="*v1.Secret" Jul 10 13:11:34.757552 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.757528 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-140-11.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-11.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Jul 10 13:11:34.759365 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.759338 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-n4shw\" is forbidden: User \"system:node:ip-10-0-140-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-11.ec2.internal' and this object" podUID="91110571-c3fa-49ac-a975-a00d7e2aee97" pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.759697 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.759670 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-140-11.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-11.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" Jul 10 13:11:34.759977 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:34.759952 2567 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-140-11.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-11.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" Jul 10 13:11:34.885337 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885308 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-root\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885414 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-textfile\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885430 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-wtmp\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885478 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-metrics-client-ca\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885492 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/91110571-c3fa-49ac-a975-a00d7e2aee97-kube-api-access-lb4cf\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-sys\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.885717 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.885585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-tls\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.930524 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.930475 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:34.986871 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.986830 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.986871 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.986873 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-metrics-client-ca\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.986890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/91110571-c3fa-49ac-a975-a00d7e2aee97-kube-api-access-lb4cf\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.986996 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-sys\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987089 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987026 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-tls\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987089 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987068 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987082 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-sys\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987095 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-root\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987147 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-root\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987196 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-textfile\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987219 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-wtmp\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987354 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987332 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-wtmp\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987416 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987400 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-metrics-client-ca\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:34.987467 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:34.987426 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-textfile\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:35.020326 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.020274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb4cf\" (UniqueName: \"kubernetes.io/projected/91110571-c3fa-49ac-a975-a00d7e2aee97-kube-api-access-lb4cf\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:35.188470 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.188435 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:35.190567 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.190542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f9d576ed-1c8b-4be1-91ad-7e672b29f503-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-zd2f7\" (UID: \"f9d576ed-1c8b-4be1-91ad-7e672b29f503\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:35.429988 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.429924 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" Jul 10 13:11:35.500127 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.500094 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:35.601262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.601233 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-zd2f7"] Jul 10 13:11:35.604065 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:35.604040 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d576ed_1c8b_4be1_91ad_7e672b29f503.slice/crio-98e0ba16119b3287500b5c174a2e7f5108cf773625351cf403db1cc521021f84 WatchSource:0}: Error finding container 98e0ba16119b3287500b5c174a2e7f5108cf773625351cf403db1cc521021f84: Status 404 returned error can't find the container with id 98e0ba16119b3287500b5c174a2e7f5108cf773625351cf403db1cc521021f84 Jul 10 13:11:35.847490 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.847459 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:35.913437 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.913413 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 13:11:35.919471 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:35.919445 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-tls\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:35.987456 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:35.987431 2567 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition Jul 10 13:11:35.987456 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:35.987445 2567 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition Jul 10 13:11:35.987586 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:35.987491 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-kube-rbac-proxy-config podName:91110571-c3fa-49ac-a975-a00d7e2aee97 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:36.487475918 +0000 UTC m=+77.203853522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-kube-rbac-proxy-config") pod "node-exporter-n4shw" (UID: "91110571-c3fa-49ac-a975-a00d7e2aee97") : failed to sync secret cache: timed out waiting for the condition Jul 10 13:11:35.987586 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:11:35.987505 2567 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-accelerators-collector-config podName:91110571-c3fa-49ac-a975-a00d7e2aee97 nodeName:}" failed. No retries permitted until 2026-07-10 13:11:36.487498149 +0000 UTC m=+77.203875753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-accelerators-collector-config") pod "node-exporter-n4shw" (UID: "91110571-c3fa-49ac-a975-a00d7e2aee97") : failed to sync configmap cache: timed out waiting for the condition Jul 10 13:11:36.118926 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.118872 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t6hvq\"" Jul 10 13:11:36.122784 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.122762 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:36.184436 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.184417 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 13:11:36.259495 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.259466 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 13:11:36.279667 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.279638 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" event={"ID":"f9d576ed-1c8b-4be1-91ad-7e672b29f503","Type":"ContainerStarted","Data":"98e0ba16119b3287500b5c174a2e7f5108cf773625351cf403db1cc521021f84"} Jul 10 13:11:36.324434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.324406 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:36.499784 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.499756 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:36.500225 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.499828 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:36.500289 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.500271 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-accelerators-collector-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:36.502121 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.502101 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/91110571-c3fa-49ac-a975-a00d7e2aee97-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n4shw\" (UID: \"91110571-c3fa-49ac-a975-a00d7e2aee97\") " pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:36.524079 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.524052 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:36.575116 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:36.575095 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n4shw" Jul 10 13:11:36.585314 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:36.585295 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91110571_c3fa_49ac_a975_a00d7e2aee97.slice/crio-e2499d3a3a4c7bbcd3a84835e27b0b93fe50631f28f985c560b4f8487c58d408 WatchSource:0}: Error finding container e2499d3a3a4c7bbcd3a84835e27b0b93fe50631f28f985c560b4f8487c58d408: Status 404 returned error can't find the container with id e2499d3a3a4c7bbcd3a84835e27b0b93fe50631f28f985c560b4f8487c58d408 Jul 10 13:11:37.284434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.284402 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4shw" event={"ID":"91110571-c3fa-49ac-a975-a00d7e2aee97","Type":"ContainerStarted","Data":"e2499d3a3a4c7bbcd3a84835e27b0b93fe50631f28f985c560b4f8487c58d408"} Jul 10 13:11:37.286410 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.286382 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" event={"ID":"f9d576ed-1c8b-4be1-91ad-7e672b29f503","Type":"ContainerStarted","Data":"ac89f904c328a1a79b7228a2f042552a8a04e98e6b4d8528fd340e455f2dd99a"} Jul 10 13:11:37.901836 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.901805 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:37.950710 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.950685 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c89c6669-58qj7"] Jul 10 13:11:37.953342 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.953325 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:37.959009 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.958989 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 13:11:37.959089 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.958989 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 13:11:37.959308 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.959287 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-gp9alaam74fg\"" Jul 10 13:11:37.959398 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.959314 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 13:11:37.959398 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.959326 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 13:11:37.960202 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.960180 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 13:11:37.960289 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.960223 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n6qk7\"" Jul 10 13:11:37.982078 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:37.982028 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c89c6669-58qj7"] Jul 10 13:11:38.010251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010226 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010334 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010255 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgdhl\" (UniqueName: \"kubernetes.io/projected/340b9c25-1e72-47c4-9815-a45f663607ae-kube-api-access-pgdhl\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010334 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010360 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/340b9c25-1e72-47c4-9815-a45f663607ae-metrics-client-ca\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010390 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010495 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-grpc-tls\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.010495 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.010479 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-tls\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.029498 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.029474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:38.110978 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.110952 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/340b9c25-1e72-47c4-9815-a45f663607ae-metrics-client-ca\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111063 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.110985 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111063 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111001 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111063 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-grpc-tls\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111063 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111049 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-tls\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111094 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111118 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgdhl\" (UniqueName: \"kubernetes.io/projected/340b9c25-1e72-47c4-9815-a45f663607ae-kube-api-access-pgdhl\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111164 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.111789 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.111765 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/340b9c25-1e72-47c4-9815-a45f663607ae-metrics-client-ca\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.113952 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.113927 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-tls\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.114051 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.114025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.114144 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.114127 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-grpc-tls\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.114193 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.114146 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.114193 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.114164 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.114397 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.114380 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/340b9c25-1e72-47c4-9815-a45f663607ae-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.116070 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.116052 2567 patch_prober.go:28] interesting pod/image-registry-6fd5697954-ngqvd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 13:11:38.116121 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.116087 2567 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" podUID="8aed2557-b2e1-499b-807e-4d61ad1f81ef" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 13:11:38.120338 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.120321 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgdhl\" (UniqueName: \"kubernetes.io/projected/340b9c25-1e72-47c4-9815-a45f663607ae-kube-api-access-pgdhl\") pod \"thanos-querier-c89c6669-58qj7\" (UID: \"340b9c25-1e72-47c4-9815-a45f663607ae\") " pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.261904 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.261823 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:38.291566 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.291534 2567 generic.go:356] "Generic (PLEG): container finished" podID="91110571-c3fa-49ac-a975-a00d7e2aee97" containerID="208b708b21c6fd357d2d50fd805cb32a434318ae640b15d3073c57978f4e908e" exitCode=0 Jul 10 13:11:38.291679 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.291622 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4shw" event={"ID":"91110571-c3fa-49ac-a975-a00d7e2aee97","Type":"ContainerDied","Data":"208b708b21c6fd357d2d50fd805cb32a434318ae640b15d3073c57978f4e908e"} Jul 10 13:11:38.294039 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.294013 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" event={"ID":"f9d576ed-1c8b-4be1-91ad-7e672b29f503","Type":"ContainerStarted","Data":"91c88a229a35e3a6de5c3b2dda780cd595717365d348fa62d4223f63a0f875b9"} Jul 10 13:11:38.294122 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.294059 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" event={"ID":"f9d576ed-1c8b-4be1-91ad-7e672b29f503","Type":"ContainerStarted","Data":"808020978a4e1ae7f67d641bc17329ac1ef3a633bfa09546a4956e2d7a404f79"} Jul 10 13:11:38.366413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.366352 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-zd2f7" podStartSLOduration=2.78068717 podStartE2EDuration="4.366329691s" totalImagesPullingTime="1.585642521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:38.36362729 +0000 UTC m=+79.080004920" watchObservedRunningTime="2026-07-10 13:11:38.366329691 +0000 UTC m=+79.082707319" Jul 10 13:11:38.366524 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.366469 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:38.429266 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.429243 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c89c6669-58qj7"] Jul 10 13:11:38.433295 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:38.433266 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340b9c25_1e72_47c4_9815_a45f663607ae.slice/crio-171c6af52f2150780bbba03c3647d35ca5639e5a4b4dc759a756cbaddb54593d WatchSource:0}: Error finding container 171c6af52f2150780bbba03c3647d35ca5639e5a4b4dc759a756cbaddb54593d: Status 404 returned error can't find the container with id 171c6af52f2150780bbba03c3647d35ca5639e5a4b4dc759a756cbaddb54593d Jul 10 13:11:38.522983 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.522909 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:38.753168 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.753138 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:38.845996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:38.845911 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:39.075276 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.075231 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs"] Jul 10 13:11:39.075276 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.075249 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:39.077806 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.077791 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.083618 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.083595 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 13:11:39.083735 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.083597 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-kqfht\"" Jul 10 13:11:39.093996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.093974 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs"] Jul 10 13:11:39.221602 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.221578 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e3743f48-b170-4b48-bdd8-72328f25ea78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5l6cs\" (UID: \"e3743f48-b170-4b48-bdd8-72328f25ea78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.221764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.221639 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b76vq\" (UniqueName: \"kubernetes.io/projected/e3743f48-b170-4b48-bdd8-72328f25ea78-kube-api-access-b76vq\") pod \"monitoring-plugin-5f68df7f58-5l6cs\" (UID: \"e3743f48-b170-4b48-bdd8-72328f25ea78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.301799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.301756 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"171c6af52f2150780bbba03c3647d35ca5639e5a4b4dc759a756cbaddb54593d"} Jul 10 13:11:39.304435 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.304407 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4shw" event={"ID":"91110571-c3fa-49ac-a975-a00d7e2aee97","Type":"ContainerStarted","Data":"06507cbe30c4bda10782cd5d03a0943158507ef8f4e810fb5da7883028582dcc"} Jul 10 13:11:39.304561 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.304473 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n4shw" event={"ID":"91110571-c3fa-49ac-a975-a00d7e2aee97","Type":"ContainerStarted","Data":"9369a8851d8f578c8f4788b1c4742c14ffe7ed49a5075bb8debe43985fae726f"} Jul 10 13:11:39.322600 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.322576 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e3743f48-b170-4b48-bdd8-72328f25ea78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5l6cs\" (UID: \"e3743f48-b170-4b48-bdd8-72328f25ea78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.322703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.322637 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b76vq\" (UniqueName: \"kubernetes.io/projected/e3743f48-b170-4b48-bdd8-72328f25ea78-kube-api-access-b76vq\") pod \"monitoring-plugin-5f68df7f58-5l6cs\" (UID: \"e3743f48-b170-4b48-bdd8-72328f25ea78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.324993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.324971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e3743f48-b170-4b48-bdd8-72328f25ea78-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-5l6cs\" (UID: \"e3743f48-b170-4b48-bdd8-72328f25ea78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.328111 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.328089 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:39.337167 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.337143 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b76vq\" (UniqueName: \"kubernetes.io/projected/e3743f48-b170-4b48-bdd8-72328f25ea78-kube-api-access-b76vq\") pod \"monitoring-plugin-5f68df7f58-5l6cs\" (UID: \"e3743f48-b170-4b48-bdd8-72328f25ea78\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.345979 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.345939 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n4shw" podStartSLOduration=4.147940779 podStartE2EDuration="5.345926288s" totalImagesPullingTime="1.133231378s" totalInitContainerRuntime="64.754131ms" isStatefulPod=true podCreationTimestamp="2026-07-10 13:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:39.344867558 +0000 UTC m=+80.061245180" watchObservedRunningTime="2026-07-10 13:11:39.345926288 +0000 UTC m=+80.062303913" Jul 10 13:11:39.386569 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.386543 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:39.517668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.517593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:39.530438 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.530413 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs"] Jul 10 13:11:39.533027 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:39.533003 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3743f48_b170_4b48_bdd8_72328f25ea78.slice/crio-022597d80194a4abeee6fabb32fe0d241de522bf00b365390c02db6b74ca2a8b WatchSource:0}: Error finding container 022597d80194a4abeee6fabb32fe0d241de522bf00b365390c02db6b74ca2a8b: Status 404 returned error can't find the container with id 022597d80194a4abeee6fabb32fe0d241de522bf00b365390c02db6b74ca2a8b Jul 10 13:11:39.658609 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.658578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:39.666361 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.666334 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp"] Jul 10 13:11:39.669542 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.669524 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.677781 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.677754 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 13:11:39.678288 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.678267 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 13:11:39.678818 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.678793 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-86gkm\"" Jul 10 13:11:39.678818 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.678810 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 13:11:39.679011 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.678810 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ec8q5ne327oqq\"" Jul 10 13:11:39.679011 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.678793 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 13:11:39.689875 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.689839 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp"] Jul 10 13:11:39.726425 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726400 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5270714f-bf33-4505-9016-9ce0bd512888-audit-log\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.726546 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726474 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-secret-metrics-server-client-certs\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.726546 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-secret-metrics-server-tls\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.726546 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726541 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5270714f-bf33-4505-9016-9ce0bd512888-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.726722 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726610 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqh8\" (UniqueName: \"kubernetes.io/projected/5270714f-bf33-4505-9016-9ce0bd512888-kube-api-access-msqh8\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.726722 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726668 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5270714f-bf33-4505-9016-9ce0bd512888-metrics-server-audit-profiles\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.726722 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.726685 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-client-ca-bundle\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827710 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827636 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-client-ca-bundle\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827710 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827676 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5270714f-bf33-4505-9016-9ce0bd512888-audit-log\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827725 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-secret-metrics-server-client-certs\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827754 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-secret-metrics-server-tls\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827783 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5270714f-bf33-4505-9016-9ce0bd512888-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827814 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqh8\" (UniqueName: \"kubernetes.io/projected/5270714f-bf33-4505-9016-9ce0bd512888-kube-api-access-msqh8\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.827905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.827883 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5270714f-bf33-4505-9016-9ce0bd512888-metrics-server-audit-profiles\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.828220 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.828196 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5270714f-bf33-4505-9016-9ce0bd512888-audit-log\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.828641 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.828617 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5270714f-bf33-4505-9016-9ce0bd512888-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.828768 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.828750 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5270714f-bf33-4505-9016-9ce0bd512888-metrics-server-audit-profiles\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.830279 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.830259 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-secret-metrics-server-client-certs\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.830370 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.830303 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-client-ca-bundle\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.830433 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.830405 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5270714f-bf33-4505-9016-9ce0bd512888-secret-metrics-server-tls\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.838080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.838053 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqh8\" (UniqueName: \"kubernetes.io/projected/5270714f-bf33-4505-9016-9ce0bd512888-kube-api-access-msqh8\") pod \"metrics-server-5fb48b6c7b-pzfvp\" (UID: \"5270714f-bf33-4505-9016-9ce0bd512888\") " pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:39.980058 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:39.980035 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:40.143013 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:40.142882 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp"] Jul 10 13:11:40.146093 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:11:40.146057 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5270714f_bf33_4505_9016_9ce0bd512888.slice/crio-1fdf152f182aeb75a440c7afcfc81e04d400681574c159031e695385ee79db34 WatchSource:0}: Error finding container 1fdf152f182aeb75a440c7afcfc81e04d400681574c159031e695385ee79db34: Status 404 returned error can't find the container with id 1fdf152f182aeb75a440c7afcfc81e04d400681574c159031e695385ee79db34 Jul 10 13:11:40.309742 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:40.309708 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" event={"ID":"e3743f48-b170-4b48-bdd8-72328f25ea78","Type":"ContainerStarted","Data":"022597d80194a4abeee6fabb32fe0d241de522bf00b365390c02db6b74ca2a8b"} Jul 10 13:11:40.310826 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:40.310802 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" event={"ID":"5270714f-bf33-4505-9016-9ce0bd512888","Type":"ContainerStarted","Data":"1fdf152f182aeb75a440c7afcfc81e04d400681574c159031e695385ee79db34"} Jul 10 13:11:41.209152 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:41.209029 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6fd5697954-ngqvd" Jul 10 13:11:41.593274 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:41.593239 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:41.766018 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:41.765939 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:41.890027 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:41.889994 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:42.320595 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.320563 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"63fc2370535525a2a55bacfb7b76141209c5120eb8d2394440b6b378487def0d"} Jul 10 13:11:42.321064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.320645 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"5cc7c48fd87390122fda398bd0bbe3e2bf32501b3be710b2535917643521c8a5"} Jul 10 13:11:42.321064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.320695 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"8d5d825e272a201b3229de6009e889987e9cd08437492f8199226c23a5adffdc"} Jul 10 13:11:42.322212 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.322178 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" event={"ID":"e3743f48-b170-4b48-bdd8-72328f25ea78","Type":"ContainerStarted","Data":"3dcc4fed627cfd35f0009a27d3fc81c1bae7f97f05123b1fd00651f35a593efd"} Jul 10 13:11:42.322404 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.322385 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:42.327930 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.327907 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" Jul 10 13:11:42.342780 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:42.342727 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-5l6cs" podStartSLOduration=1.5548451559999998 podStartE2EDuration="3.342711745s" totalImagesPullingTime="1.787866589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:42.342460117 +0000 UTC m=+83.058837752" watchObservedRunningTime="2026-07-10 13:11:42.342711745 +0000 UTC m=+83.059089372" Jul 10 13:11:43.329299 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.329266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"8888620035d08e19963a0e45f411b2870e33b0af827012598051b64d4c5c3880"} Jul 10 13:11:43.329689 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.329330 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"6b8dcbe1e57f915f9ff170811e2eef2f2f8e7524a1c20f70fa283666b7af969d"} Jul 10 13:11:43.329689 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.329370 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" event={"ID":"340b9c25-1e72-47c4-9815-a45f663607ae","Type":"ContainerStarted","Data":"9746a06ea7ef3d8712a41b34b62f6456b463c1442a6025db3c1b5ae3cc2ee2ad"} Jul 10 13:11:43.329689 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.329416 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:43.330902 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.330878 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" event={"ID":"5270714f-bf33-4505-9016-9ce0bd512888","Type":"ContainerStarted","Data":"2f97eeb6d9d7affcf3def4e843a3d32fd96861e23de7a4adac96019d946c8e9d"} Jul 10 13:11:43.363956 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.363914 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" podStartSLOduration=2.474268927 podStartE2EDuration="6.363900581s" totalImagesPullingTime="3.889631654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:43.360868249 +0000 UTC m=+84.077245895" watchObservedRunningTime="2026-07-10 13:11:43.363900581 +0000 UTC m=+84.080278207" Jul 10 13:11:43.392894 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.392833 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" podStartSLOduration=1.9867725200000002 podStartE2EDuration="4.392820977s" totalImagesPullingTime="2.406048457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:11:43.39146254 +0000 UTC m=+84.107840165" watchObservedRunningTime="2026-07-10 13:11:43.392820977 +0000 UTC m=+84.109198603" Jul 10 13:11:43.534046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.534020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:43.716718 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:43.716667 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:44.198737 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:44.198707 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:44.313199 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:44.313172 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:44.503692 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:44.503620 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:44.608172 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:44.608143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:49.339511 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:49.339486 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c89c6669-58qj7" Jul 10 13:11:50.701780 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:50.701746 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:50.772027 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:50.771998 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:51.179287 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:51.179212 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:51.287990 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:51.287957 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:51.396421 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:51.396395 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:51.820646 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:51.820618 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:51.919677 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:51.919653 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:52.027017 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:52.026991 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:52.126456 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:52.126396 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:52.640127 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:52.640093 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:52.741891 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:52.741857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:59.466364 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:59.466329 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:59.552249 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:59.552203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:11:59.980819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:59.980792 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:11:59.980984 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:11:59.980857 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:12:04.274342 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:04.274308 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tmb52" Jul 10 13:12:06.153757 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:06.153730 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:12:14.468087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:14.468051 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:12:14.568417 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:14.568375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:12:19.985366 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:19.985337 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:12:19.989143 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:19.989116 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5fb48b6c7b-pzfvp" Jul 10 13:12:54.360145 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:54.360114 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:12:54.867271 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:54.867238 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:12:59.440485 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.440452 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-698d6d76c5-qzlpf"] Jul 10 13:12:59.443032 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.443018 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.445992 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.445967 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-f7pwt\"" Jul 10 13:12:59.446113 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.445997 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 13:12:59.446244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.446224 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 13:12:59.446289 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.446274 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 13:12:59.446384 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.446370 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 13:12:59.446455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.446441 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 13:12:59.451447 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.451426 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 13:12:59.457545 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.457528 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-698d6d76c5-qzlpf"] Jul 10 13:12:59.470264 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470243 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-telemeter-trusted-ca-bundle\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470295 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-metrics-client-ca\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-secret-telemeter-client\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470327 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6qt\" (UniqueName: \"kubernetes.io/projected/5a616786-9fdb-478c-a17c-12795174e260-kube-api-access-fn6qt\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470475 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470406 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-federate-client-tls\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470475 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470444 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-telemeter-client-tls\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.470475 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.470469 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-serving-certs-ca-bundle\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.571781 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571749 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.571781 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571779 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-telemeter-trusted-ca-bundle\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571801 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-metrics-client-ca\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571819 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-secret-telemeter-client\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571838 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6qt\" (UniqueName: \"kubernetes.io/projected/5a616786-9fdb-478c-a17c-12795174e260-kube-api-access-fn6qt\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571912 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-federate-client-tls\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571945 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-telemeter-client-tls\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.571975 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-serving-certs-ca-bundle\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.572599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-serving-certs-ca-bundle\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572725 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.572599 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-metrics-client-ca\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.572812 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.572786 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a616786-9fdb-478c-a17c-12795174e260-telemeter-trusted-ca-bundle\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.574521 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.574496 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-telemeter-client-tls\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.574595 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.574579 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-federate-client-tls\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.574736 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.574719 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-secret-telemeter-client\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.574827 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.574805 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a616786-9fdb-478c-a17c-12795174e260-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.593958 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.593936 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6qt\" (UniqueName: \"kubernetes.io/projected/5a616786-9fdb-478c-a17c-12795174e260-kube-api-access-fn6qt\") pod \"telemeter-client-698d6d76c5-qzlpf\" (UID: \"5a616786-9fdb-478c-a17c-12795174e260\") " pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.752924 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.752905 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" Jul 10 13:12:59.896729 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:12:59.896697 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-698d6d76c5-qzlpf"] Jul 10 13:12:59.900220 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:12:59.900193 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a616786_9fdb_478c_a17c_12795174e260.slice/crio-a7780ab01ac5d8949436c5a912a971ace0b80a57088acec3c858bd6b5f1d6a99 WatchSource:0}: Error finding container a7780ab01ac5d8949436c5a912a971ace0b80a57088acec3c858bd6b5f1d6a99: Status 404 returned error can't find the container with id a7780ab01ac5d8949436c5a912a971ace0b80a57088acec3c858bd6b5f1d6a99 Jul 10 13:13:00.595913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:00.595874 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" event={"ID":"5a616786-9fdb-478c-a17c-12795174e260","Type":"ContainerStarted","Data":"a7780ab01ac5d8949436c5a912a971ace0b80a57088acec3c858bd6b5f1d6a99"} Jul 10 13:13:02.603674 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:02.603646 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" event={"ID":"5a616786-9fdb-478c-a17c-12795174e260","Type":"ContainerStarted","Data":"41064cb32a6b011356174ccbb709b0e8c3fc564a7eba963c87a167e25e76b2fd"} Jul 10 13:13:04.613413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:04.613373 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" event={"ID":"5a616786-9fdb-478c-a17c-12795174e260","Type":"ContainerStarted","Data":"3b6e1405136f35558499d9404dc7d06e15a864bb097fcf0960119474ef0ae641"} Jul 10 13:13:04.613801 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:04.613438 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" event={"ID":"5a616786-9fdb-478c-a17c-12795174e260","Type":"ContainerStarted","Data":"ba41609caa867728c190a8a1d4a835e28b30d6efb79f46c2d88577dc40eb07c5"} Jul 10 13:13:04.654361 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:04.654308 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-698d6d76c5-qzlpf" podStartSLOduration=2.051053786 podStartE2EDuration="5.65429287s" totalImagesPullingTime="3.603239084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:12:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:13:04.65040685 +0000 UTC m=+165.366784489" watchObservedRunningTime="2026-07-10 13:13:04.65429287 +0000 UTC m=+165.370670496" Jul 10 13:13:05.967138 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:05.967106 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:06.109759 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:06.109723 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:08.149881 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:08.149809 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:09.312568 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:09.312535 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:09.724392 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:09.724364 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:09.818250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:09.818226 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:16.686250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:16.686163 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:16.827873 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:16.827830 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:21.552457 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:21.552420 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:21.662737 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:21.662708 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:27.458573 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:27.458544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:27.616020 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:27.615989 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:27.870076 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:27.869999 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:29.559584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:29.559549 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:13:29.722120 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:13:29.722089 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:01.257511 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:01.257474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:07.368243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:07.368204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:07.468634 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:07.468600 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:08.437465 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.437433 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m"] Jul 10 13:14:08.443125 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.443101 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.445986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.445963 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 13:14:08.446204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.446185 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nwcmn\"" Jul 10 13:14:08.447046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.447029 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 13:14:08.452027 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.452002 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m"] Jul 10 13:14:08.536485 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.536456 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.536625 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.536493 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.536625 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.536517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw9k9\" (UniqueName: \"kubernetes.io/projected/3052bc71-ff22-49aa-b020-447b017ad43c-kube-api-access-fw9k9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.637051 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.637029 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.637137 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.637058 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.637137 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.637080 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw9k9\" (UniqueName: \"kubernetes.io/projected/3052bc71-ff22-49aa-b020-447b017ad43c-kube-api-access-fw9k9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.637413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.637396 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.637449 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.637417 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.646457 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.646427 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw9k9\" (UniqueName: \"kubernetes.io/projected/3052bc71-ff22-49aa-b020-447b017ad43c-kube-api-access-fw9k9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.752463 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.752437 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:08.873879 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:08.873857 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m"] Jul 10 13:14:08.876299 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:14:08.876272 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3052bc71_ff22_49aa_b020_447b017ad43c.slice/crio-460f7cbde3b6d7784f1fbbb9dd0573bc1b308309beecb669c5c4df7c39c200f4 WatchSource:0}: Error finding container 460f7cbde3b6d7784f1fbbb9dd0573bc1b308309beecb669c5c4df7c39c200f4: Status 404 returned error can't find the container with id 460f7cbde3b6d7784f1fbbb9dd0573bc1b308309beecb669c5c4df7c39c200f4 Jul 10 13:14:09.847220 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:09.847180 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" event={"ID":"3052bc71-ff22-49aa-b020-447b017ad43c","Type":"ContainerStarted","Data":"460f7cbde3b6d7784f1fbbb9dd0573bc1b308309beecb669c5c4df7c39c200f4"} Jul 10 13:14:19.885030 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:19.884971 2567 generic.go:356] "Generic (PLEG): container finished" podID="3052bc71-ff22-49aa-b020-447b017ad43c" containerID="8ec1e058d7c1867e28e3b605c101ccf0111d834446fbf6f12418078044ea9bd5" exitCode=0 Jul 10 13:14:19.885030 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:19.885008 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" event={"ID":"3052bc71-ff22-49aa-b020-447b017ad43c","Type":"ContainerDied","Data":"8ec1e058d7c1867e28e3b605c101ccf0111d834446fbf6f12418078044ea9bd5"} Jul 10 13:14:22.897102 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:22.897059 2567 generic.go:356] "Generic (PLEG): container finished" podID="3052bc71-ff22-49aa-b020-447b017ad43c" containerID="c08e2e3c2acd0c234a8d619db8b24130879701a4ed1a62a71c8c01927fd8632d" exitCode=0 Jul 10 13:14:22.897448 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:22.897114 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" event={"ID":"3052bc71-ff22-49aa-b020-447b017ad43c","Type":"ContainerDied","Data":"c08e2e3c2acd0c234a8d619db8b24130879701a4ed1a62a71c8c01927fd8632d"} Jul 10 13:14:31.931278 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:31.931189 2567 generic.go:356] "Generic (PLEG): container finished" podID="3052bc71-ff22-49aa-b020-447b017ad43c" containerID="934291546a8e0ed7086fca30aac29fbd4b9d61fbecea6f127840b1fe873bf047" exitCode=0 Jul 10 13:14:31.931278 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:31.931249 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" event={"ID":"3052bc71-ff22-49aa-b020-447b017ad43c","Type":"ContainerDied","Data":"934291546a8e0ed7086fca30aac29fbd4b9d61fbecea6f127840b1fe873bf047"} Jul 10 13:14:33.052895 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.052871 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:33.135718 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.135680 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-bundle\") pod \"3052bc71-ff22-49aa-b020-447b017ad43c\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " Jul 10 13:14:33.135886 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.135760 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw9k9\" (UniqueName: \"kubernetes.io/projected/3052bc71-ff22-49aa-b020-447b017ad43c-kube-api-access-fw9k9\") pod \"3052bc71-ff22-49aa-b020-447b017ad43c\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " Jul 10 13:14:33.136340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.136307 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-bundle" (OuterVolumeSpecName: "bundle") pod "3052bc71-ff22-49aa-b020-447b017ad43c" (UID: "3052bc71-ff22-49aa-b020-447b017ad43c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:14:33.137800 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.137776 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3052bc71-ff22-49aa-b020-447b017ad43c-kube-api-access-fw9k9" (OuterVolumeSpecName: "kube-api-access-fw9k9") pod "3052bc71-ff22-49aa-b020-447b017ad43c" (UID: "3052bc71-ff22-49aa-b020-447b017ad43c"). InnerVolumeSpecName "kube-api-access-fw9k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:14:33.236923 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.236896 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-util\") pod \"3052bc71-ff22-49aa-b020-447b017ad43c\" (UID: \"3052bc71-ff22-49aa-b020-447b017ad43c\") " Jul 10 13:14:33.237114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.237099 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:14:33.237175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.237120 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw9k9\" (UniqueName: \"kubernetes.io/projected/3052bc71-ff22-49aa-b020-447b017ad43c-kube-api-access-fw9k9\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:14:33.241317 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.241295 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-util" (OuterVolumeSpecName: "util") pod "3052bc71-ff22-49aa-b020-447b017ad43c" (UID: "3052bc71-ff22-49aa-b020-447b017ad43c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:14:33.337683 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.337648 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3052bc71-ff22-49aa-b020-447b017ad43c-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:14:33.940768 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.940738 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" event={"ID":"3052bc71-ff22-49aa-b020-447b017ad43c","Type":"ContainerDied","Data":"460f7cbde3b6d7784f1fbbb9dd0573bc1b308309beecb669c5c4df7c39c200f4"} Jul 10 13:14:33.940942 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.940780 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540pnl4m" Jul 10 13:14:33.940942 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:33.940812 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="460f7cbde3b6d7784f1fbbb9dd0573bc1b308309beecb669c5c4df7c39c200f4" Jul 10 13:14:40.031363 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:40.031327 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:41.206885 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.206786 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:41.847799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.847766 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw"] Jul 10 13:14:41.848051 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848039 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="pull" Jul 10 13:14:41.848108 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848052 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="pull" Jul 10 13:14:41.848108 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848068 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="util" Jul 10 13:14:41.848108 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848073 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="util" Jul 10 13:14:41.848108 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848080 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="extract" Jul 10 13:14:41.848108 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848085 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="extract" Jul 10 13:14:41.848271 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.848137 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3052bc71-ff22-49aa-b020-447b017ad43c" containerName="extract" Jul 10 13:14:41.849950 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.849935 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:41.853682 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.853657 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 13:14:41.853816 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.853796 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hmg4z\"" Jul 10 13:14:41.854094 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.854069 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:14:41.870938 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.870914 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw"] Jul 10 13:14:41.898972 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.898937 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbtbw\" (UniqueName: \"kubernetes.io/projected/38091c0f-7b0d-498a-8d74-05c239ac46c8-kube-api-access-dbtbw\") pod \"cert-manager-operator-controller-manager-55c444bf86-p4fdw\" (UID: \"38091c0f-7b0d-498a-8d74-05c239ac46c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:41.899172 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.898987 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38091c0f-7b0d-498a-8d74-05c239ac46c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p4fdw\" (UID: \"38091c0f-7b0d-498a-8d74-05c239ac46c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:41.999888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.999857 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbtbw\" (UniqueName: \"kubernetes.io/projected/38091c0f-7b0d-498a-8d74-05c239ac46c8-kube-api-access-dbtbw\") pod \"cert-manager-operator-controller-manager-55c444bf86-p4fdw\" (UID: \"38091c0f-7b0d-498a-8d74-05c239ac46c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:41.999888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:41.999890 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38091c0f-7b0d-498a-8d74-05c239ac46c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p4fdw\" (UID: \"38091c0f-7b0d-498a-8d74-05c239ac46c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:42.000199 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:42.000184 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38091c0f-7b0d-498a-8d74-05c239ac46c8-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p4fdw\" (UID: \"38091c0f-7b0d-498a-8d74-05c239ac46c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:42.025969 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:42.025946 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbtbw\" (UniqueName: \"kubernetes.io/projected/38091c0f-7b0d-498a-8d74-05c239ac46c8-kube-api-access-dbtbw\") pod \"cert-manager-operator-controller-manager-55c444bf86-p4fdw\" (UID: \"38091c0f-7b0d-498a-8d74-05c239ac46c8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:42.159015 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:42.158946 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" Jul 10 13:14:42.291905 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:42.291878 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw"] Jul 10 13:14:42.293767 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:14:42.293740 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38091c0f_7b0d_498a_8d74_05c239ac46c8.slice/crio-fc893d1d196203c6c5c6151a1e71cea9b333706be603f6e28f16f97381dec143 WatchSource:0}: Error finding container fc893d1d196203c6c5c6151a1e71cea9b333706be603f6e28f16f97381dec143: Status 404 returned error can't find the container with id fc893d1d196203c6c5c6151a1e71cea9b333706be603f6e28f16f97381dec143 Jul 10 13:14:42.975201 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:42.975162 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" event={"ID":"38091c0f-7b0d-498a-8d74-05c239ac46c8","Type":"ContainerStarted","Data":"fc893d1d196203c6c5c6151a1e71cea9b333706be603f6e28f16f97381dec143"} Jul 10 13:14:43.669168 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:43.669127 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:43.776252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:43.776206 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:44.984952 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:44.984915 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" event={"ID":"38091c0f-7b0d-498a-8d74-05c239ac46c8","Type":"ContainerStarted","Data":"61c7d5a843c6514111a94784254e743c651f055a875faa069068216fef610948"} Jul 10 13:14:45.036648 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:45.036595 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p4fdw" podStartSLOduration=2.233295942 podStartE2EDuration="4.036575077s" totalImagesPullingTime="1.803279135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:14:45.036544657 +0000 UTC m=+265.752922309" watchObservedRunningTime="2026-07-10 13:14:45.036575077 +0000 UTC m=+265.752952729" Jul 10 13:14:45.721747 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:45.721711 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:45.933012 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:45.932970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:46.144053 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.143974 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:46.369097 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.369061 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:46.402670 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.402595 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb"] Jul 10 13:14:46.405252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.405236 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.418893 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.418871 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 13:14:46.419007 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.418955 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nwcmn\"" Jul 10 13:14:46.420505 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.420489 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 13:14:46.421971 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:14:46.421948 2567 status_manager.go:1018] "Failed to get status for pod" err="pods \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" is forbidden: User \"system:node:ip-10-0-140-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-140-11.ec2.internal' and this object" podUID="ec50d974-4588-4549-a105-7939955d5747" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.436264 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.436239 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbcf\" (UniqueName: \"kubernetes.io/projected/ec50d974-4588-4549-a105-7939955d5747-kube-api-access-6kbcf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.436382 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.436281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.436382 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.436316 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.507811 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.507780 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb"] Jul 10 13:14:46.536824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.536792 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbcf\" (UniqueName: \"kubernetes.io/projected/ec50d974-4588-4549-a105-7939955d5747-kube-api-access-6kbcf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.536985 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.536874 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.536985 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.536926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.537250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.537231 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.537323 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.537268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.571302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.571275 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbcf\" (UniqueName: \"kubernetes.io/projected/ec50d974-4588-4549-a105-7939955d5747-kube-api-access-6kbcf\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.714213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.714178 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:46.861018 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.860983 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb"] Jul 10 13:14:46.864403 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:14:46.864366 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec50d974_4588_4549_a105_7939955d5747.slice/crio-a402efe659bafdbf8e8faa6e4ebc0d063f809fa7dca1bbf30f3b736d76f7e553 WatchSource:0}: Error finding container a402efe659bafdbf8e8faa6e4ebc0d063f809fa7dca1bbf30f3b736d76f7e553: Status 404 returned error can't find the container with id a402efe659bafdbf8e8faa6e4ebc0d063f809fa7dca1bbf30f3b736d76f7e553 Jul 10 13:14:46.994262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.994185 2567 generic.go:356] "Generic (PLEG): container finished" podID="ec50d974-4588-4549-a105-7939955d5747" containerID="615a127ac0e731e9aabd9fad37a6b0b992c5a75494414a2b0fb1a6fb4be7c72b" exitCode=0 Jul 10 13:14:46.994400 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.994272 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" event={"ID":"ec50d974-4588-4549-a105-7939955d5747","Type":"ContainerDied","Data":"615a127ac0e731e9aabd9fad37a6b0b992c5a75494414a2b0fb1a6fb4be7c72b"} Jul 10 13:14:46.994400 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:46.994327 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" event={"ID":"ec50d974-4588-4549-a105-7939955d5747","Type":"ContainerStarted","Data":"a402efe659bafdbf8e8faa6e4ebc0d063f809fa7dca1bbf30f3b736d76f7e553"} Jul 10 13:14:47.951599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:47.951573 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:48.559310 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:48.559282 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:50.007155 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:50.007124 2567 generic.go:356] "Generic (PLEG): container finished" podID="ec50d974-4588-4549-a105-7939955d5747" containerID="777f49d8b35c30df183588118f47b9ced7c493c19f30f558a1ecefaaa5df1f0e" exitCode=0 Jul 10 13:14:50.007507 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:50.007176 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" event={"ID":"ec50d974-4588-4549-a105-7939955d5747","Type":"ContainerDied","Data":"777f49d8b35c30df183588118f47b9ced7c493c19f30f558a1ecefaaa5df1f0e"} Jul 10 13:14:50.629134 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:50.629099 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:50.731645 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:50.731614 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:51.013524 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:51.013493 2567 generic.go:356] "Generic (PLEG): container finished" podID="ec50d974-4588-4549-a105-7939955d5747" containerID="6280fc3cc67f55289763892fae4025dcb7d2af79313ee1b4385077d9a48f54f1" exitCode=0 Jul 10 13:14:51.013922 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:51.013551 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" event={"ID":"ec50d974-4588-4549-a105-7939955d5747","Type":"ContainerDied","Data":"6280fc3cc67f55289763892fae4025dcb7d2af79313ee1b4385077d9a48f54f1"} Jul 10 13:14:52.137785 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.137761 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:52.185383 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.185355 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-bundle\") pod \"ec50d974-4588-4549-a105-7939955d5747\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " Jul 10 13:14:52.185531 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.185400 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kbcf\" (UniqueName: \"kubernetes.io/projected/ec50d974-4588-4549-a105-7939955d5747-kube-api-access-6kbcf\") pod \"ec50d974-4588-4549-a105-7939955d5747\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " Jul 10 13:14:52.185531 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.185452 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-util\") pod \"ec50d974-4588-4549-a105-7939955d5747\" (UID: \"ec50d974-4588-4549-a105-7939955d5747\") " Jul 10 13:14:52.185859 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.185813 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-bundle" (OuterVolumeSpecName: "bundle") pod "ec50d974-4588-4549-a105-7939955d5747" (UID: "ec50d974-4588-4549-a105-7939955d5747"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:14:52.187431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.187409 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec50d974-4588-4549-a105-7939955d5747-kube-api-access-6kbcf" (OuterVolumeSpecName: "kube-api-access-6kbcf") pod "ec50d974-4588-4549-a105-7939955d5747" (UID: "ec50d974-4588-4549-a105-7939955d5747"). InnerVolumeSpecName "kube-api-access-6kbcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:14:52.191106 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.190933 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-util" (OuterVolumeSpecName: "util") pod "ec50d974-4588-4549-a105-7939955d5747" (UID: "ec50d974-4588-4549-a105-7939955d5747"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:14:52.286893 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.286792 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:14:52.286893 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.286820 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbcf\" (UniqueName: \"kubernetes.io/projected/ec50d974-4588-4549-a105-7939955d5747-kube-api-access-6kbcf\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:14:52.286893 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:52.286829 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec50d974-4588-4549-a105-7939955d5747-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:14:53.023219 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.023188 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" event={"ID":"ec50d974-4588-4549-a105-7939955d5747","Type":"ContainerDied","Data":"a402efe659bafdbf8e8faa6e4ebc0d063f809fa7dca1bbf30f3b736d76f7e553"} Jul 10 13:14:53.023219 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.023220 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5npkb" Jul 10 13:14:53.023416 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.023231 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a402efe659bafdbf8e8faa6e4ebc0d063f809fa7dca1bbf30f3b736d76f7e553" Jul 10 13:14:53.467691 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.467662 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2dhps"] Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.467969 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="util" Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.467980 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="util" Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.467992 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="pull" Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.467997 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="pull" Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.468005 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="extract" Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.468011 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="extract" Jul 10 13:14:53.468083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.468060 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec50d974-4588-4549-a105-7939955d5747" containerName="extract" Jul 10 13:14:53.470128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.470113 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.477062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.477039 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 13:14:53.477151 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.477065 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 13:14:53.478083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.478067 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-klnt5\"" Jul 10 13:14:53.483522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.483502 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2dhps"] Jul 10 13:14:53.594473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.594434 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/112a0b7d-9625-4a34-98a5-aa935165930e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2dhps\" (UID: \"112a0b7d-9625-4a34-98a5-aa935165930e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.594473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.594477 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtgq\" (UniqueName: \"kubernetes.io/projected/112a0b7d-9625-4a34-98a5-aa935165930e-kube-api-access-qxtgq\") pod \"cert-manager-cainjector-559cdd8c87-2dhps\" (UID: \"112a0b7d-9625-4a34-98a5-aa935165930e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.694868 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.694811 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/112a0b7d-9625-4a34-98a5-aa935165930e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2dhps\" (UID: \"112a0b7d-9625-4a34-98a5-aa935165930e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.695205 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.694896 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtgq\" (UniqueName: \"kubernetes.io/projected/112a0b7d-9625-4a34-98a5-aa935165930e-kube-api-access-qxtgq\") pod \"cert-manager-cainjector-559cdd8c87-2dhps\" (UID: \"112a0b7d-9625-4a34-98a5-aa935165930e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.709320 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.709287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/112a0b7d-9625-4a34-98a5-aa935165930e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2dhps\" (UID: \"112a0b7d-9625-4a34-98a5-aa935165930e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.710770 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.710754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtgq\" (UniqueName: \"kubernetes.io/projected/112a0b7d-9625-4a34-98a5-aa935165930e-kube-api-access-qxtgq\") pod \"cert-manager-cainjector-559cdd8c87-2dhps\" (UID: \"112a0b7d-9625-4a34-98a5-aa935165930e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.795312 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.795223 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" Jul 10 13:14:53.928711 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:53.928679 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2dhps"] Jul 10 13:14:53.932633 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:14:53.932602 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod112a0b7d_9625_4a34_98a5_aa935165930e.slice/crio-ddf3e64d81bef68daa594ad8acc3925aeb8b988d1a5114e96f29046d6689ca58 WatchSource:0}: Error finding container ddf3e64d81bef68daa594ad8acc3925aeb8b988d1a5114e96f29046d6689ca58: Status 404 returned error can't find the container with id ddf3e64d81bef68daa594ad8acc3925aeb8b988d1a5114e96f29046d6689ca58 Jul 10 13:14:54.028792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:54.028757 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" event={"ID":"112a0b7d-9625-4a34-98a5-aa935165930e","Type":"ContainerStarted","Data":"ddf3e64d81bef68daa594ad8acc3925aeb8b988d1a5114e96f29046d6689ca58"} Jul 10 13:14:56.945867 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:56.945817 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:57.044754 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:57.044717 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" event={"ID":"112a0b7d-9625-4a34-98a5-aa935165930e","Type":"ContainerStarted","Data":"f44f47842778bbbadebbb3b3994de5d13daddebec799d6920eab52e0d4b74719"} Jul 10 13:14:57.045507 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:57.045472 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:14:57.060927 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:57.060887 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2dhps" podStartSLOduration=1.748787294 podStartE2EDuration="4.060871511s" totalImagesPullingTime="2.312084217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:14:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:14:57.06044893 +0000 UTC m=+277.776826585" watchObservedRunningTime="2026-07-10 13:14:57.060871511 +0000 UTC m=+277.777249138" Jul 10 13:14:57.146235 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:14:57.146204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:01.056563 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:01.056526 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:01.158323 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:01.158294 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:03.248348 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.248320 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx"] Jul 10 13:15:03.250629 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.250610 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.253423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.253399 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 13:15:03.253510 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.253440 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 13:15:03.254482 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.254462 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-gsxgg\"" Jul 10 13:15:03.260030 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.260011 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx"] Jul 10 13:15:03.366662 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.366639 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/118abeaa-67b0-432c-a763-1f8a2efd5b5d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wzhvx\" (UID: \"118abeaa-67b0-432c-a763-1f8a2efd5b5d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.366748 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.366671 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfwk\" (UniqueName: \"kubernetes.io/projected/118abeaa-67b0-432c-a763-1f8a2efd5b5d-kube-api-access-7bfwk\") pod \"openshift-lws-operator-fd8ccff4c-wzhvx\" (UID: \"118abeaa-67b0-432c-a763-1f8a2efd5b5d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.467984 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.467953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/118abeaa-67b0-432c-a763-1f8a2efd5b5d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wzhvx\" (UID: \"118abeaa-67b0-432c-a763-1f8a2efd5b5d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.468096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.467990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfwk\" (UniqueName: \"kubernetes.io/projected/118abeaa-67b0-432c-a763-1f8a2efd5b5d-kube-api-access-7bfwk\") pod \"openshift-lws-operator-fd8ccff4c-wzhvx\" (UID: \"118abeaa-67b0-432c-a763-1f8a2efd5b5d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.468352 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.468333 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/118abeaa-67b0-432c-a763-1f8a2efd5b5d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-wzhvx\" (UID: \"118abeaa-67b0-432c-a763-1f8a2efd5b5d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.476372 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.476346 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfwk\" (UniqueName: \"kubernetes.io/projected/118abeaa-67b0-432c-a763-1f8a2efd5b5d-kube-api-access-7bfwk\") pod \"openshift-lws-operator-fd8ccff4c-wzhvx\" (UID: \"118abeaa-67b0-432c-a763-1f8a2efd5b5d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.559939 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.559887 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" Jul 10 13:15:03.683784 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:03.683759 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx"] Jul 10 13:15:03.686303 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:15:03.686271 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod118abeaa_67b0_432c_a763_1f8a2efd5b5d.slice/crio-44781c6383ef9acf94ca1864b531c85a446d7daeb8024aaf202d3213a2fb31de WatchSource:0}: Error finding container 44781c6383ef9acf94ca1864b531c85a446d7daeb8024aaf202d3213a2fb31de: Status 404 returned error can't find the container with id 44781c6383ef9acf94ca1864b531c85a446d7daeb8024aaf202d3213a2fb31de Jul 10 13:15:04.071867 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:04.071825 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" event={"ID":"118abeaa-67b0-432c-a763-1f8a2efd5b5d","Type":"ContainerStarted","Data":"44781c6383ef9acf94ca1864b531c85a446d7daeb8024aaf202d3213a2fb31de"} Jul 10 13:15:07.086990 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:07.086953 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" event={"ID":"118abeaa-67b0-432c-a763-1f8a2efd5b5d","Type":"ContainerStarted","Data":"106b54badf57f8b4f5034794dec363b69693b7720a1a31c2d9d3c594ac17b5c1"} Jul 10 13:15:07.108125 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:07.104021 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-wzhvx" podStartSLOduration=1.41448106 podStartE2EDuration="4.104001519s" totalImagesPullingTime="2.689520459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:07.102811737 +0000 UTC m=+287.819189364" watchObservedRunningTime="2026-07-10 13:15:07.104001519 +0000 UTC m=+287.820379146" Jul 10 13:15:07.810310 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:07.810280 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:07.911714 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:07.911687 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:09.514272 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:09.514222 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:12.464140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.464105 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw"] Jul 10 13:15:12.466391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.466375 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.469993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.469960 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 13:15:12.470959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.470931 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 13:15:12.471158 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.471141 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nwcmn\"" Jul 10 13:15:12.480484 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.480462 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw"] Jul 10 13:15:12.532708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.532686 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s65v\" (UniqueName: \"kubernetes.io/projected/c4bea9da-9989-46b4-a874-03a8e988791c-kube-api-access-6s65v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.532820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.532739 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.532820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.532759 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.633522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.633486 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.633522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.633526 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.633667 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.633564 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s65v\" (UniqueName: \"kubernetes.io/projected/c4bea9da-9989-46b4-a874-03a8e988791c-kube-api-access-6s65v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.633952 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.633931 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.633952 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.633943 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.643917 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.643892 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s65v\" (UniqueName: \"kubernetes.io/projected/c4bea9da-9989-46b4-a874-03a8e988791c-kube-api-access-6s65v\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.777304 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.777252 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:12.903975 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:12.903949 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw"] Jul 10 13:15:12.905971 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:15:12.905941 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bea9da_9989_46b4_a874_03a8e988791c.slice/crio-e96550ffc3abc8b515b26f40cbadea25503e20c3a3a82660f14b6c2fce8082be WatchSource:0}: Error finding container e96550ffc3abc8b515b26f40cbadea25503e20c3a3a82660f14b6c2fce8082be: Status 404 returned error can't find the container with id e96550ffc3abc8b515b26f40cbadea25503e20c3a3a82660f14b6c2fce8082be Jul 10 13:15:13.111681 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:13.111623 2567 generic.go:356] "Generic (PLEG): container finished" podID="c4bea9da-9989-46b4-a874-03a8e988791c" containerID="01e16e81fbc435321dac7bd1d078936d86ba7afdbbb2dcc48fd4cceb673c662e" exitCode=0 Jul 10 13:15:13.111792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:13.111711 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" event={"ID":"c4bea9da-9989-46b4-a874-03a8e988791c","Type":"ContainerDied","Data":"01e16e81fbc435321dac7bd1d078936d86ba7afdbbb2dcc48fd4cceb673c662e"} Jul 10 13:15:13.111792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:13.111773 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" event={"ID":"c4bea9da-9989-46b4-a874-03a8e988791c","Type":"ContainerStarted","Data":"e96550ffc3abc8b515b26f40cbadea25503e20c3a3a82660f14b6c2fce8082be"} Jul 10 13:15:14.119677 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:14.119650 2567 generic.go:356] "Generic (PLEG): container finished" podID="c4bea9da-9989-46b4-a874-03a8e988791c" containerID="a46660acdca9c4fd000ac2ff906e06ecbf7ccbec4ab98c7bb6f642a4d21bb075" exitCode=0 Jul 10 13:15:14.119981 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:14.119716 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" event={"ID":"c4bea9da-9989-46b4-a874-03a8e988791c","Type":"ContainerDied","Data":"a46660acdca9c4fd000ac2ff906e06ecbf7ccbec4ab98c7bb6f642a4d21bb075"} Jul 10 13:15:15.125649 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:15.125618 2567 generic.go:356] "Generic (PLEG): container finished" podID="c4bea9da-9989-46b4-a874-03a8e988791c" containerID="327a73d3d8f51bfbfabc898cf0e2b8f574019ec52e80159c026486224ced9356" exitCode=0 Jul 10 13:15:15.126023 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:15.125675 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" event={"ID":"c4bea9da-9989-46b4-a874-03a8e988791c","Type":"ContainerDied","Data":"327a73d3d8f51bfbfabc898cf0e2b8f574019ec52e80159c026486224ced9356"} Jul 10 13:15:16.253164 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.253143 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:16.360593 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.360565 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-util\") pod \"c4bea9da-9989-46b4-a874-03a8e988791c\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " Jul 10 13:15:16.360729 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.360616 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-bundle\") pod \"c4bea9da-9989-46b4-a874-03a8e988791c\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " Jul 10 13:15:16.360729 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.360644 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s65v\" (UniqueName: \"kubernetes.io/projected/c4bea9da-9989-46b4-a874-03a8e988791c-kube-api-access-6s65v\") pod \"c4bea9da-9989-46b4-a874-03a8e988791c\" (UID: \"c4bea9da-9989-46b4-a874-03a8e988791c\") " Jul 10 13:15:16.361311 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.361288 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-bundle" (OuterVolumeSpecName: "bundle") pod "c4bea9da-9989-46b4-a874-03a8e988791c" (UID: "c4bea9da-9989-46b4-a874-03a8e988791c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:15:16.362788 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.362766 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bea9da-9989-46b4-a874-03a8e988791c-kube-api-access-6s65v" (OuterVolumeSpecName: "kube-api-access-6s65v") pod "c4bea9da-9989-46b4-a874-03a8e988791c" (UID: "c4bea9da-9989-46b4-a874-03a8e988791c"). InnerVolumeSpecName "kube-api-access-6s65v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:15:16.365972 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.365945 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-util" (OuterVolumeSpecName: "util") pod "c4bea9da-9989-46b4-a874-03a8e988791c" (UID: "c4bea9da-9989-46b4-a874-03a8e988791c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:15:16.462086 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.462067 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:15:16.462175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.462087 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s65v\" (UniqueName: \"kubernetes.io/projected/c4bea9da-9989-46b4-a874-03a8e988791c-kube-api-access-6s65v\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:15:16.462175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:16.462097 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c4bea9da-9989-46b4-a874-03a8e988791c-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:15:17.136290 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:17.136266 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" Jul 10 13:15:17.136452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:17.136262 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afhlmjw" event={"ID":"c4bea9da-9989-46b4-a874-03a8e988791c","Type":"ContainerDied","Data":"e96550ffc3abc8b515b26f40cbadea25503e20c3a3a82660f14b6c2fce8082be"} Jul 10 13:15:17.136452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:17.136380 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e96550ffc3abc8b515b26f40cbadea25503e20c3a3a82660f14b6c2fce8082be" Jul 10 13:15:19.744298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:19.744273 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:15:19.744738 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:19.744455 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:15:19.748675 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:19.748656 2567 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 13:15:23.060744 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:23.060715 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:23.163314 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:23.163283 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:23.660740 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:23.660701 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.008357 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.008331 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.133797 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.133764 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.183257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183233 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x"] Jul 10 13:15:24.183501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183490 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="util" Jul 10 13:15:24.183543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183503 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="util" Jul 10 13:15:24.183543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183518 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="pull" Jul 10 13:15:24.183543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183523 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="pull" Jul 10 13:15:24.183543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183530 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="extract" Jul 10 13:15:24.183543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183535 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="extract" Jul 10 13:15:24.183730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.183597 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4bea9da-9989-46b4-a874-03a8e988791c" containerName="extract" Jul 10 13:15:24.189359 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.189338 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.192732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.192702 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 13:15:24.193494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.193471 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tq2hx\"" Jul 10 13:15:24.193649 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.193623 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 13:15:24.193726 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.193708 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 13:15:24.193825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.193802 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 13:15:24.219773 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.219737 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x"] Jul 10 13:15:24.321119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.321054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6816528-0fe2-474e-9ea8-8a201c8be438-manifests\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.321119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.321085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6816528-0fe2-474e-9ea8-8a201c8be438-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.321119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.321107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d8mn\" (UniqueName: \"kubernetes.io/projected/b6816528-0fe2-474e-9ea8-8a201c8be438-kube-api-access-8d8mn\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.321353 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.321188 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6816528-0fe2-474e-9ea8-8a201c8be438-tmp\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.321353 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.321274 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6816528-0fe2-474e-9ea8-8a201c8be438-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.422409 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.422383 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6816528-0fe2-474e-9ea8-8a201c8be438-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.422527 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.422418 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6816528-0fe2-474e-9ea8-8a201c8be438-manifests\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.422527 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.422437 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6816528-0fe2-474e-9ea8-8a201c8be438-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.422527 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.422457 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d8mn\" (UniqueName: \"kubernetes.io/projected/b6816528-0fe2-474e-9ea8-8a201c8be438-kube-api-access-8d8mn\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.422527 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.422508 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6816528-0fe2-474e-9ea8-8a201c8be438-tmp\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.422892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.422866 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b6816528-0fe2-474e-9ea8-8a201c8be438-manifests\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.424419 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.424388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.424816 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.424788 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b6816528-0fe2-474e-9ea8-8a201c8be438-tmp\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.424979 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.424961 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b6816528-0fe2-474e-9ea8-8a201c8be438-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.425138 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.425119 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b6816528-0fe2-474e-9ea8-8a201c8be438-webhook-cert\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.430789 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.430770 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d8mn\" (UniqueName: \"kubernetes.io/projected/b6816528-0fe2-474e-9ea8-8a201c8be438-kube-api-access-8d8mn\") pod \"opendatahub-operator-controller-manager-9ff96f8f8-2qd5x\" (UID: \"b6816528-0fe2-474e-9ea8-8a201c8be438\") " pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.503191 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.503169 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:24.528288 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.528262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.630959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.630929 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.651235 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.651215 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x"] Jul 10 13:15:24.653023 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:15:24.652999 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6816528_0fe2_474e_9ea8_8a201c8be438.slice/crio-0ed16805fc1608998d68fc932e96fd82fbdd01251243986d22578b90392e0cc7 WatchSource:0}: Error finding container 0ed16805fc1608998d68fc932e96fd82fbdd01251243986d22578b90392e0cc7: Status 404 returned error can't find the container with id 0ed16805fc1608998d68fc932e96fd82fbdd01251243986d22578b90392e0cc7 Jul 10 13:15:24.654890 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.654875 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:15:24.793046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.793005 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.888357 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.888257 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:24.986604 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:24.986577 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:25.169223 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:25.169157 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" event={"ID":"b6816528-0fe2-474e-9ea8-8a201c8be438","Type":"ContainerStarted","Data":"0ed16805fc1608998d68fc932e96fd82fbdd01251243986d22578b90392e0cc7"} Jul 10 13:15:28.186244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:28.186169 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" event={"ID":"b6816528-0fe2-474e-9ea8-8a201c8be438","Type":"ContainerStarted","Data":"3ed8a3bbb58c0a894b2d18770c345bcd721e8fe68e842172abbacb17ed0f364d"} Jul 10 13:15:29.191898 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:29.191797 2567 generic.go:356] "Generic (PLEG): container finished" podID="b6816528-0fe2-474e-9ea8-8a201c8be438" containerID="3ed8a3bbb58c0a894b2d18770c345bcd721e8fe68e842172abbacb17ed0f364d" exitCode=0 Jul 10 13:15:29.191898 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:29.191875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" event={"ID":"b6816528-0fe2-474e-9ea8-8a201c8be438","Type":"ContainerDied","Data":"3ed8a3bbb58c0a894b2d18770c345bcd721e8fe68e842172abbacb17ed0f364d"} Jul 10 13:15:30.198381 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:30.198332 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" event={"ID":"b6816528-0fe2-474e-9ea8-8a201c8be438","Type":"ContainerStarted","Data":"3eee18f51e1a0079a84de7822a06de925a05a1977c8b51a58923a93847a537bc"} Jul 10 13:15:30.198971 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:30.198418 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:30.245926 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:30.245868 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" podStartSLOduration=1.5357154 podStartE2EDuration="6.245834211s" totalImagesPullingTime="3.637587715s" totalInitContainerRuntime="1.072531096s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:30.242792932 +0000 UTC m=+310.959170559" watchObservedRunningTime="2026-07-10 13:15:30.245834211 +0000 UTC m=+310.962211836" Jul 10 13:15:35.370224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:35.370190 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:35.499529 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:35.499497 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:39.760221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:39.760181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:39.910990 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:39.910962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:41.206494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.206465 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9ff96f8f8-2qd5x" Jul 10 13:15:41.746165 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.746137 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx"] Jul 10 13:15:41.749227 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.749207 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.754643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.754624 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 13:15:41.754762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.754643 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2d9bg\"" Jul 10 13:15:41.754762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.754626 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 13:15:41.754999 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.754986 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 13:15:41.812080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.812053 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx"] Jul 10 13:15:41.867839 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.867811 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4781f44d-5278-46fc-8ec8-6873630c39ef-manager-config\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.867968 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.867922 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4781f44d-5278-46fc-8ec8-6873630c39ef-cert\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.867968 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.867959 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4781f44d-5278-46fc-8ec8-6873630c39ef-metrics-cert\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.868070 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.867995 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t87c\" (UniqueName: \"kubernetes.io/projected/4781f44d-5278-46fc-8ec8-6873630c39ef-kube-api-access-6t87c\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.968620 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.968589 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4781f44d-5278-46fc-8ec8-6873630c39ef-manager-config\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.968767 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.968652 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4781f44d-5278-46fc-8ec8-6873630c39ef-cert\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.968767 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.968688 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4781f44d-5278-46fc-8ec8-6873630c39ef-metrics-cert\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.968767 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.968728 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t87c\" (UniqueName: \"kubernetes.io/projected/4781f44d-5278-46fc-8ec8-6873630c39ef-kube-api-access-6t87c\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.969224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.969198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4781f44d-5278-46fc-8ec8-6873630c39ef-manager-config\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.971008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.970988 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4781f44d-5278-46fc-8ec8-6873630c39ef-cert\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.971104 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.971078 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4781f44d-5278-46fc-8ec8-6873630c39ef-metrics-cert\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:41.986825 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:41.986800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t87c\" (UniqueName: \"kubernetes.io/projected/4781f44d-5278-46fc-8ec8-6873630c39ef-kube-api-access-6t87c\") pod \"lws-controller-manager-7cffcb8bcf-vt6mx\" (UID: \"4781f44d-5278-46fc-8ec8-6873630c39ef\") " pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:42.058872 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:42.058810 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:42.193099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:42.193071 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx"] Jul 10 13:15:42.195648 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:15:42.195623 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4781f44d_5278_46fc_8ec8_6873630c39ef.slice/crio-cf9477cdea978a64511c43ff38f32a288e0e285ae2924fd476829afc49082b65 WatchSource:0}: Error finding container cf9477cdea978a64511c43ff38f32a288e0e285ae2924fd476829afc49082b65: Status 404 returned error can't find the container with id cf9477cdea978a64511c43ff38f32a288e0e285ae2924fd476829afc49082b65 Jul 10 13:15:42.252347 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:42.252314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" event={"ID":"4781f44d-5278-46fc-8ec8-6873630c39ef","Type":"ContainerStarted","Data":"cf9477cdea978a64511c43ff38f32a288e0e285ae2924fd476829afc49082b65"} Jul 10 13:15:44.263139 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:44.263098 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" event={"ID":"4781f44d-5278-46fc-8ec8-6873630c39ef","Type":"ContainerStarted","Data":"9db4fc53efad82aa113c32672eb108332ef5b657e3fa34c09f9679f47504b7e6"} Jul 10 13:15:44.263526 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:44.263227 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:44.294335 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:44.294286 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" podStartSLOduration=1.5655443340000001 podStartE2EDuration="3.294270804s" totalImagesPullingTime="1.72872647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:15:44.293381492 +0000 UTC m=+325.009759137" watchObservedRunningTime="2026-07-10 13:15:44.294270804 +0000 UTC m=+325.010648429" Jul 10 13:15:44.963394 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:44.963362 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:45.186939 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:45.186904 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:45.905266 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:45.905234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:46.131972 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:46.131937 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:46.512394 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:46.512366 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:46.812506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:46.812417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:46.986401 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:46.986372 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:47.186195 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:47.186110 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:47.590420 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:47.590383 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:47.810185 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:47.810141 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:48.261363 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:48.261330 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:48.396586 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:48.396551 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:49.427860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:49.427808 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:49.557666 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:49.557618 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:51.409261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:51.409226 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:54.508185 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:54.508158 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:54.657286 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:54.657256 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:55.271560 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:55.271534 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cffcb8bcf-vt6mx" Jul 10 13:15:56.013998 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.013965 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b"] Jul 10 13:15:56.018834 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.018814 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.034903 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.034882 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 13:15:56.035004 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.034934 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 13:15:56.035709 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.035693 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mctm7\"" Jul 10 13:15:56.055835 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.055816 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 13:15:56.077966 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.077946 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b"] Jul 10 13:15:56.093298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093278 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093389 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093389 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093329 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093389 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093350 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093449 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30b42e9-6fb7-406d-8933-31b41682c8e6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093517 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093540 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdqd7\" (UniqueName: \"kubernetes.io/projected/a30b42e9-6fb7-406d-8933-31b41682c8e6-kube-api-access-vdqd7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.093632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.093564 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.143628 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.143606 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:56.194396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194367 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194515 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194424 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30b42e9-6fb7-406d-8933-31b41682c8e6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194515 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194515 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194482 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdqd7\" (UniqueName: \"kubernetes.io/projected/a30b42e9-6fb7-406d-8933-31b41682c8e6-kube-api-access-vdqd7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194519 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194549 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194595 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194617 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194654 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.194962 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.194904 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.195126 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.195080 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.195217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.195169 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.195378 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.195350 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.195482 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.195393 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a30b42e9-6fb7-406d-8933-31b41682c8e6-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.196973 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.196952 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.197316 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.197294 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.209373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.209351 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdqd7\" (UniqueName: \"kubernetes.io/projected/a30b42e9-6fb7-406d-8933-31b41682c8e6-kube-api-access-vdqd7\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.219350 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.219327 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a30b42e9-6fb7-406d-8933-31b41682c8e6-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4zf96b\" (UID: \"a30b42e9-6fb7-406d-8933-31b41682c8e6\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.334156 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.334082 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:15:56.516759 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.516736 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b"] Jul 10 13:15:56.518340 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:15:56.518317 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda30b42e9_6fb7_406d_8933_31b41682c8e6.slice/crio-80ff70315b46e9939a448f1122c2a173395fcac5a129878d2bcdb93ec526d396 WatchSource:0}: Error finding container 80ff70315b46e9939a448f1122c2a173395fcac5a129878d2bcdb93ec526d396: Status 404 returned error can't find the container with id 80ff70315b46e9939a448f1122c2a173395fcac5a129878d2bcdb93ec526d396 Jul 10 13:15:56.543117 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.543068 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:56.680251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.680180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:56.681223 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.681202 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5"] Jul 10 13:15:56.684461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.684445 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.691301 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.691282 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 13:15:56.691831 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.691813 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 13:15:56.691947 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.691876 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8tq4d\"" Jul 10 13:15:56.691947 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.691909 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 13:15:56.702956 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.702925 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5"] Jul 10 13:15:56.800129 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800101 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e82c6e60-f99b-4307-950c-46701a51a262-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.800243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800138 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.800243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82c6e60-f99b-4307-950c-46701a51a262-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.800243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800230 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrw5\" (UniqueName: \"kubernetes.io/projected/e82c6e60-f99b-4307-950c-46701a51a262-kube-api-access-bkrw5\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.800345 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800275 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.800345 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82c6e60-f99b-4307-950c-46701a51a262-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.800345 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.800338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901454 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82c6e60-f99b-4307-950c-46701a51a262-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901516 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrw5\" (UniqueName: \"kubernetes.io/projected/e82c6e60-f99b-4307-950c-46701a51a262-kube-api-access-bkrw5\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901548 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901594 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82c6e60-f99b-4307-950c-46701a51a262-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901628 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901887 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901696 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e82c6e60-f99b-4307-950c-46701a51a262-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.901887 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.901724 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.902555 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.902512 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e82c6e60-f99b-4307-950c-46701a51a262-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.904045 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.904012 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e82c6e60-f99b-4307-950c-46701a51a262-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.904171 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.904115 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.904328 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.904309 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.904815 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.904796 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e82c6e60-f99b-4307-950c-46701a51a262-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.926092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.926073 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrw5\" (UniqueName: \"kubernetes.io/projected/e82c6e60-f99b-4307-950c-46701a51a262-kube-api-access-bkrw5\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.926092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.926084 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e82c6e60-f99b-4307-950c-46701a51a262-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-zmcd5\" (UID: \"e82c6e60-f99b-4307-950c-46701a51a262\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:56.994465 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:56.994440 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:15:57.165486 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:57.165418 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5"] Jul 10 13:15:57.168022 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:15:57.167992 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82c6e60_f99b_4307_950c_46701a51a262.slice/crio-4430a27440aff74bb315ed390199b3c5bc43a0cd364ded5107de93921cd4f242 WatchSource:0}: Error finding container 4430a27440aff74bb315ed390199b3c5bc43a0cd364ded5107de93921cd4f242: Status 404 returned error can't find the container with id 4430a27440aff74bb315ed390199b3c5bc43a0cd364ded5107de93921cd4f242 Jul 10 13:15:57.325977 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:57.325889 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" event={"ID":"a30b42e9-6fb7-406d-8933-31b41682c8e6","Type":"ContainerStarted","Data":"80ff70315b46e9939a448f1122c2a173395fcac5a129878d2bcdb93ec526d396"} Jul 10 13:15:57.327378 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:57.327352 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" event={"ID":"e82c6e60-f99b-4307-950c-46701a51a262","Type":"ContainerStarted","Data":"4430a27440aff74bb315ed390199b3c5bc43a0cd364ded5107de93921cd4f242"} Jul 10 13:15:58.189358 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:58.189292 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:58.454788 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:58.454726 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:58.649492 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:58.649417 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:59.821221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:59.821183 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:15:59.954893 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:15:59.954839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:16:00.154168 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.154087 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:16:00.164461 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.164423 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:16:00.164562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.164481 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:16:00.164562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.164505 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:16:00.164562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.164529 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:16:00.164562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.164543 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:16:00.345222 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.345178 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" event={"ID":"e82c6e60-f99b-4307-950c-46701a51a262","Type":"ContainerStarted","Data":"26cc3022c7764e88239399c341c5682213805baf92025ec88b2000afe4096711"} Jul 10 13:16:00.345503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.345472 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:16:00.346833 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.346809 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" event={"ID":"a30b42e9-6fb7-406d-8933-31b41682c8e6","Type":"ContainerStarted","Data":"8a474acd19e969d4c4b8517ed6eecd8aeaabb7acbee0f44ff305acd2bb6777cf"} Jul 10 13:16:00.388972 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.388920 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" podStartSLOduration=1.39671009 podStartE2EDuration="4.388903556s" totalImagesPullingTime="2.992193466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:16:00.386771486 +0000 UTC m=+341.103149112" watchObservedRunningTime="2026-07-10 13:16:00.388903556 +0000 UTC m=+341.105281182" Jul 10 13:16:00.441598 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:00.441511 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" podStartSLOduration=1.797629826 podStartE2EDuration="5.441496438s" totalImagesPullingTime="3.643866612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:15:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:16:00.440190391 +0000 UTC m=+341.156568018" watchObservedRunningTime="2026-07-10 13:16:00.441496438 +0000 UTC m=+341.157874064" Jul 10 13:16:01.334434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:01.334400 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:16:01.339075 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:01.339052 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:16:01.352073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:01.352041 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:16:01.352747 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:01.352724 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4zf96b" Jul 10 13:16:01.353391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:01.353372 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-zmcd5" Jul 10 13:16:47.036237 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:47.036143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:16:47.139702 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:47.139670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:16:47.250302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:16:47.250271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:03.346991 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.346955 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc"] Jul 10 13:17:03.349543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.349522 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.352569 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.352541 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 13:17:03.355865 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.352664 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fd6r5\"" Jul 10 13:17:03.355865 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.353291 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 13:17:03.360967 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.360941 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc"] Jul 10 13:17:03.425093 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.425070 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtkkx\" (UniqueName: \"kubernetes.io/projected/02a4aba3-abee-4b36-bb31-5273afe623d0-kube-api-access-qtkkx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.425213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.425104 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.425213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.425135 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.525622 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.525586 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.525729 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.525656 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtkkx\" (UniqueName: \"kubernetes.io/projected/02a4aba3-abee-4b36-bb31-5273afe623d0-kube-api-access-qtkkx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.525729 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.525680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.526041 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.526022 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.526086 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.526025 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.533621 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.533597 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtkkx\" (UniqueName: \"kubernetes.io/projected/02a4aba3-abee-4b36-bb31-5273afe623d0-kube-api-access-qtkkx\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.664462 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.664389 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:03.754962 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.754930 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9"] Jul 10 13:17:03.758251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.758233 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.767932 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.767834 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9"] Jul 10 13:17:03.801112 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.801085 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc"] Jul 10 13:17:03.802592 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:17:03.802569 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a4aba3_abee_4b36_bb31_5273afe623d0.slice/crio-ba68c4155523b16e859075eb6424c3bb9a910dc30796ed12f8fb3cf2a6e2af8f WatchSource:0}: Error finding container ba68c4155523b16e859075eb6424c3bb9a910dc30796ed12f8fb3cf2a6e2af8f: Status 404 returned error can't find the container with id ba68c4155523b16e859075eb6424c3bb9a910dc30796ed12f8fb3cf2a6e2af8f Jul 10 13:17:03.828467 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.828440 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.828550 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.828504 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.828593 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.828563 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7wf7\" (UniqueName: \"kubernetes.io/projected/09b4ecc1-9b42-4c81-adc7-e378b8674627-kube-api-access-l7wf7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.929158 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.929103 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.929158 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.929150 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7wf7\" (UniqueName: \"kubernetes.io/projected/09b4ecc1-9b42-4c81-adc7-e378b8674627-kube-api-access-l7wf7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.929311 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.929191 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.929460 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.929440 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.929460 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.929457 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:03.939088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:03.939067 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7wf7\" (UniqueName: \"kubernetes.io/projected/09b4ecc1-9b42-4c81-adc7-e378b8674627-kube-api-access-l7wf7\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:04.070209 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.070187 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:04.161447 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.161418 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4"] Jul 10 13:17:04.164322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.164296 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.175773 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.175746 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4"] Jul 10 13:17:04.200557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.200534 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9"] Jul 10 13:17:04.202264 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:17:04.202245 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b4ecc1_9b42_4c81_adc7_e378b8674627.slice/crio-e9ff49ee57b52cb0fc8623f05575069a3264d3c04a71685a57eaf7961f81909a WatchSource:0}: Error finding container e9ff49ee57b52cb0fc8623f05575069a3264d3c04a71685a57eaf7961f81909a: Status 404 returned error can't find the container with id e9ff49ee57b52cb0fc8623f05575069a3264d3c04a71685a57eaf7961f81909a Jul 10 13:17:04.231720 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.231697 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.231805 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.231751 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.231873 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.231818 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf8bd\" (UniqueName: \"kubernetes.io/projected/b2576174-c9b6-4b39-baac-05982b9e5f4e-kube-api-access-zf8bd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.332608 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.332585 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.332711 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.332642 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.332711 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.332671 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf8bd\" (UniqueName: \"kubernetes.io/projected/b2576174-c9b6-4b39-baac-05982b9e5f4e-kube-api-access-zf8bd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.333028 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.333007 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.333070 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.333018 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.343956 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.343935 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf8bd\" (UniqueName: \"kubernetes.io/projected/b2576174-c9b6-4b39-baac-05982b9e5f4e-kube-api-access-zf8bd\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.475966 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.475915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:04.605127 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.605096 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t"] Jul 10 13:17:04.608303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.608279 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.626322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.626305 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4"] Jul 10 13:17:04.629155 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:17:04.629118 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2576174_c9b6_4b39_baac_05982b9e5f4e.slice/crio-c8372d934d572d177deaa646bc9c563213b687d8d31a31e9df668a767d87298c WatchSource:0}: Error finding container c8372d934d572d177deaa646bc9c563213b687d8d31a31e9df668a767d87298c: Status 404 returned error can't find the container with id c8372d934d572d177deaa646bc9c563213b687d8d31a31e9df668a767d87298c Jul 10 13:17:04.632041 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.632020 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t"] Jul 10 13:17:04.634882 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.634857 2567 generic.go:356] "Generic (PLEG): container finished" podID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerID="6618d2cb872c99bf51fa67a4640dad423514b31512b6e84030f0e04fe819f225" exitCode=0 Jul 10 13:17:04.635012 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.634981 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" event={"ID":"09b4ecc1-9b42-4c81-adc7-e378b8674627","Type":"ContainerDied","Data":"6618d2cb872c99bf51fa67a4640dad423514b31512b6e84030f0e04fe819f225"} Jul 10 13:17:04.635088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.635025 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" event={"ID":"09b4ecc1-9b42-4c81-adc7-e378b8674627","Type":"ContainerStarted","Data":"e9ff49ee57b52cb0fc8623f05575069a3264d3c04a71685a57eaf7961f81909a"} Jul 10 13:17:04.637424 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.637403 2567 generic.go:356] "Generic (PLEG): container finished" podID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerID="3949cfc7d2d6f3a5f0893c23883aabd350a8327295cae4d0a4c9b3747e2f79c5" exitCode=0 Jul 10 13:17:04.637557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.637463 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" event={"ID":"02a4aba3-abee-4b36-bb31-5273afe623d0","Type":"ContainerDied","Data":"3949cfc7d2d6f3a5f0893c23883aabd350a8327295cae4d0a4c9b3747e2f79c5"} Jul 10 13:17:04.637557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.637494 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" event={"ID":"02a4aba3-abee-4b36-bb31-5273afe623d0","Type":"ContainerStarted","Data":"ba68c4155523b16e859075eb6424c3bb9a910dc30796ed12f8fb3cf2a6e2af8f"} Jul 10 13:17:04.735300 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.735271 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.735402 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.735347 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ncq2\" (UniqueName: \"kubernetes.io/projected/a977dd51-c671-4b36-891d-2ada964af0ea-kube-api-access-7ncq2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.735402 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.735373 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.836354 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.836332 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.836472 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.836384 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ncq2\" (UniqueName: \"kubernetes.io/projected/a977dd51-c671-4b36-891d-2ada964af0ea-kube-api-access-7ncq2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.836472 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.836406 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.836734 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.836712 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.836799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.836732 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.846579 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.846559 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ncq2\" (UniqueName: \"kubernetes.io/projected/a977dd51-c671-4b36-891d-2ada964af0ea-kube-api-access-7ncq2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:04.916901 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:04.916880 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:05.052550 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.052526 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t"] Jul 10 13:17:05.054526 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:17:05.054495 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda977dd51_c671_4b36_891d_2ada964af0ea.slice/crio-1080a37fee692b033a6e85746e32519878d4b1fa9e8d894ba7115c1bf0904cb5 WatchSource:0}: Error finding container 1080a37fee692b033a6e85746e32519878d4b1fa9e8d894ba7115c1bf0904cb5: Status 404 returned error can't find the container with id 1080a37fee692b033a6e85746e32519878d4b1fa9e8d894ba7115c1bf0904cb5 Jul 10 13:17:05.643964 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.643880 2567 generic.go:356] "Generic (PLEG): container finished" podID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerID="c80bf3daf1ecaf89fe56c24876bd47be241ff130b73541d6186c8c127f18aa21" exitCode=0 Jul 10 13:17:05.644335 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.643969 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" event={"ID":"09b4ecc1-9b42-4c81-adc7-e378b8674627","Type":"ContainerDied","Data":"c80bf3daf1ecaf89fe56c24876bd47be241ff130b73541d6186c8c127f18aa21"} Jul 10 13:17:05.645506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.645370 2567 generic.go:356] "Generic (PLEG): container finished" podID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerID="743b209d551e1ae6e018cb2834abba1c4f9e94543b87d5a4de8c45c38cd5939a" exitCode=0 Jul 10 13:17:05.645506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.645448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" event={"ID":"b2576174-c9b6-4b39-baac-05982b9e5f4e","Type":"ContainerDied","Data":"743b209d551e1ae6e018cb2834abba1c4f9e94543b87d5a4de8c45c38cd5939a"} Jul 10 13:17:05.645506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.645489 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" event={"ID":"b2576174-c9b6-4b39-baac-05982b9e5f4e","Type":"ContainerStarted","Data":"c8372d934d572d177deaa646bc9c563213b687d8d31a31e9df668a767d87298c"} Jul 10 13:17:05.647639 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.647591 2567 generic.go:356] "Generic (PLEG): container finished" podID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerID="10ad6fd03688cba912d5897f061abaca9c7f8155bff34051bf15faf87c7f7622" exitCode=0 Jul 10 13:17:05.647712 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.647672 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" event={"ID":"02a4aba3-abee-4b36-bb31-5273afe623d0","Type":"ContainerDied","Data":"10ad6fd03688cba912d5897f061abaca9c7f8155bff34051bf15faf87c7f7622"} Jul 10 13:17:05.649357 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.649335 2567 generic.go:356] "Generic (PLEG): container finished" podID="a977dd51-c671-4b36-891d-2ada964af0ea" containerID="5e087bee56d95055b20d0b03eae31be4fd78a5cca9b70ea6c407c011a6e9a059" exitCode=0 Jul 10 13:17:05.649463 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.649377 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" event={"ID":"a977dd51-c671-4b36-891d-2ada964af0ea","Type":"ContainerDied","Data":"5e087bee56d95055b20d0b03eae31be4fd78a5cca9b70ea6c407c011a6e9a059"} Jul 10 13:17:05.649463 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:05.649400 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" event={"ID":"a977dd51-c671-4b36-891d-2ada964af0ea","Type":"ContainerStarted","Data":"1080a37fee692b033a6e85746e32519878d4b1fa9e8d894ba7115c1bf0904cb5"} Jul 10 13:17:06.656408 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.656380 2567 generic.go:356] "Generic (PLEG): container finished" podID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerID="cd35c28abd80d25824c0caeabae3f31b33f73ac46ad06e922d85cefc5d5684df" exitCode=0 Jul 10 13:17:06.656744 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.656461 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" event={"ID":"09b4ecc1-9b42-4c81-adc7-e378b8674627","Type":"ContainerDied","Data":"cd35c28abd80d25824c0caeabae3f31b33f73ac46ad06e922d85cefc5d5684df"} Jul 10 13:17:06.658399 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.658374 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" event={"ID":"b2576174-c9b6-4b39-baac-05982b9e5f4e","Type":"ContainerStarted","Data":"e943fb530a8dfbe8a69ef054984bf97bdb1bd28f02d75c02830fe9157bafb6a6"} Jul 10 13:17:06.660526 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.660502 2567 generic.go:356] "Generic (PLEG): container finished" podID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerID="e46374273d025dcef95a5ea8ebda2f1940c5b93a967797710e2bb41360e1dd5c" exitCode=0 Jul 10 13:17:06.660638 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.660578 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" event={"ID":"02a4aba3-abee-4b36-bb31-5273afe623d0","Type":"ContainerDied","Data":"e46374273d025dcef95a5ea8ebda2f1940c5b93a967797710e2bb41360e1dd5c"} Jul 10 13:17:06.662584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.662562 2567 generic.go:356] "Generic (PLEG): container finished" podID="a977dd51-c671-4b36-891d-2ada964af0ea" containerID="e97a02933fec2658f711895d817b8714edeb7df644ff3d391e18aed233f21fc3" exitCode=0 Jul 10 13:17:06.662665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:06.662601 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" event={"ID":"a977dd51-c671-4b36-891d-2ada964af0ea","Type":"ContainerDied","Data":"e97a02933fec2658f711895d817b8714edeb7df644ff3d391e18aed233f21fc3"} Jul 10 13:17:07.670113 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.670075 2567 generic.go:356] "Generic (PLEG): container finished" podID="a977dd51-c671-4b36-891d-2ada964af0ea" containerID="0bc71bd2f9d3ac63de83c0848784bd57e9d5fcac03947a8a4556d2028b1a88cd" exitCode=0 Jul 10 13:17:07.670505 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.670148 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" event={"ID":"a977dd51-c671-4b36-891d-2ada964af0ea","Type":"ContainerDied","Data":"0bc71bd2f9d3ac63de83c0848784bd57e9d5fcac03947a8a4556d2028b1a88cd"} Jul 10 13:17:07.672031 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.672007 2567 generic.go:356] "Generic (PLEG): container finished" podID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerID="e943fb530a8dfbe8a69ef054984bf97bdb1bd28f02d75c02830fe9157bafb6a6" exitCode=0 Jul 10 13:17:07.672160 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.672095 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" event={"ID":"b2576174-c9b6-4b39-baac-05982b9e5f4e","Type":"ContainerDied","Data":"e943fb530a8dfbe8a69ef054984bf97bdb1bd28f02d75c02830fe9157bafb6a6"} Jul 10 13:17:07.809026 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.809006 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:07.861244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.861217 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7wf7\" (UniqueName: \"kubernetes.io/projected/09b4ecc1-9b42-4c81-adc7-e378b8674627-kube-api-access-l7wf7\") pod \"09b4ecc1-9b42-4c81-adc7-e378b8674627\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " Jul 10 13:17:07.861340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.861265 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-util\") pod \"09b4ecc1-9b42-4c81-adc7-e378b8674627\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " Jul 10 13:17:07.861340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.861295 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-bundle\") pod \"09b4ecc1-9b42-4c81-adc7-e378b8674627\" (UID: \"09b4ecc1-9b42-4c81-adc7-e378b8674627\") " Jul 10 13:17:07.861881 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.861829 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-bundle" (OuterVolumeSpecName: "bundle") pod "09b4ecc1-9b42-4c81-adc7-e378b8674627" (UID: "09b4ecc1-9b42-4c81-adc7-e378b8674627"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:07.863341 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.863320 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09b4ecc1-9b42-4c81-adc7-e378b8674627-kube-api-access-l7wf7" (OuterVolumeSpecName: "kube-api-access-l7wf7") pod "09b4ecc1-9b42-4c81-adc7-e378b8674627" (UID: "09b4ecc1-9b42-4c81-adc7-e378b8674627"). InnerVolumeSpecName "kube-api-access-l7wf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:17:07.866570 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.866547 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-util" (OuterVolumeSpecName: "util") pod "09b4ecc1-9b42-4c81-adc7-e378b8674627" (UID: "09b4ecc1-9b42-4c81-adc7-e378b8674627"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:07.899534 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.899511 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:07.961923 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.961905 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-bundle\") pod \"02a4aba3-abee-4b36-bb31-5273afe623d0\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " Jul 10 13:17:07.961999 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.961972 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-util\") pod \"02a4aba3-abee-4b36-bb31-5273afe623d0\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " Jul 10 13:17:07.962055 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.962004 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtkkx\" (UniqueName: \"kubernetes.io/projected/02a4aba3-abee-4b36-bb31-5273afe623d0-kube-api-access-qtkkx\") pod \"02a4aba3-abee-4b36-bb31-5273afe623d0\" (UID: \"02a4aba3-abee-4b36-bb31-5273afe623d0\") " Jul 10 13:17:07.962135 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.962123 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:07.962198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.962136 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/09b4ecc1-9b42-4c81-adc7-e378b8674627-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:07.962198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.962144 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7wf7\" (UniqueName: \"kubernetes.io/projected/09b4ecc1-9b42-4c81-adc7-e378b8674627-kube-api-access-l7wf7\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:07.962417 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.962397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-bundle" (OuterVolumeSpecName: "bundle") pod "02a4aba3-abee-4b36-bb31-5273afe623d0" (UID: "02a4aba3-abee-4b36-bb31-5273afe623d0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:07.964015 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.963994 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a4aba3-abee-4b36-bb31-5273afe623d0-kube-api-access-qtkkx" (OuterVolumeSpecName: "kube-api-access-qtkkx") pod "02a4aba3-abee-4b36-bb31-5273afe623d0" (UID: "02a4aba3-abee-4b36-bb31-5273afe623d0"). InnerVolumeSpecName "kube-api-access-qtkkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:17:07.967593 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:07.967561 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-util" (OuterVolumeSpecName: "util") pod "02a4aba3-abee-4b36-bb31-5273afe623d0" (UID: "02a4aba3-abee-4b36-bb31-5273afe623d0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:08.062819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.062794 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:08.062819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.062815 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtkkx\" (UniqueName: \"kubernetes.io/projected/02a4aba3-abee-4b36-bb31-5273afe623d0-kube-api-access-qtkkx\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:08.062945 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.062826 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02a4aba3-abee-4b36-bb31-5273afe623d0-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:08.678362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.678329 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" Jul 10 13:17:08.678362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.678347 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc" event={"ID":"02a4aba3-abee-4b36-bb31-5273afe623d0","Type":"ContainerDied","Data":"ba68c4155523b16e859075eb6424c3bb9a910dc30796ed12f8fb3cf2a6e2af8f"} Jul 10 13:17:08.678802 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.678387 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba68c4155523b16e859075eb6424c3bb9a910dc30796ed12f8fb3cf2a6e2af8f" Jul 10 13:17:08.680267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.680236 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" event={"ID":"09b4ecc1-9b42-4c81-adc7-e378b8674627","Type":"ContainerDied","Data":"e9ff49ee57b52cb0fc8623f05575069a3264d3c04a71685a57eaf7961f81909a"} Jul 10 13:17:08.680410 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.680276 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9ff49ee57b52cb0fc8623f05575069a3264d3c04a71685a57eaf7961f81909a" Jul 10 13:17:08.680410 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.680246 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9" Jul 10 13:17:08.682471 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.682444 2567 generic.go:356] "Generic (PLEG): container finished" podID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerID="da618a3c19dcf3d6c6088a17c39e930c263fd821ca54243fdded1c3b2005ab74" exitCode=0 Jul 10 13:17:08.682624 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.682536 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" event={"ID":"b2576174-c9b6-4b39-baac-05982b9e5f4e","Type":"ContainerDied","Data":"da618a3c19dcf3d6c6088a17c39e930c263fd821ca54243fdded1c3b2005ab74"} Jul 10 13:17:08.804730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.804710 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:08.868522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.868491 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ncq2\" (UniqueName: \"kubernetes.io/projected/a977dd51-c671-4b36-891d-2ada964af0ea-kube-api-access-7ncq2\") pod \"a977dd51-c671-4b36-891d-2ada964af0ea\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " Jul 10 13:17:08.868656 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.868530 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-bundle\") pod \"a977dd51-c671-4b36-891d-2ada964af0ea\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " Jul 10 13:17:08.868656 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.868550 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-util\") pod \"a977dd51-c671-4b36-891d-2ada964af0ea\" (UID: \"a977dd51-c671-4b36-891d-2ada964af0ea\") " Jul 10 13:17:08.869070 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.869039 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-bundle" (OuterVolumeSpecName: "bundle") pod "a977dd51-c671-4b36-891d-2ada964af0ea" (UID: "a977dd51-c671-4b36-891d-2ada964af0ea"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:08.870467 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.870445 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a977dd51-c671-4b36-891d-2ada964af0ea-kube-api-access-7ncq2" (OuterVolumeSpecName: "kube-api-access-7ncq2") pod "a977dd51-c671-4b36-891d-2ada964af0ea" (UID: "a977dd51-c671-4b36-891d-2ada964af0ea"). InnerVolumeSpecName "kube-api-access-7ncq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:17:08.874222 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.874195 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-util" (OuterVolumeSpecName: "util") pod "a977dd51-c671-4b36-891d-2ada964af0ea" (UID: "a977dd51-c671-4b36-891d-2ada964af0ea"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:08.969938 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.969912 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ncq2\" (UniqueName: \"kubernetes.io/projected/a977dd51-c671-4b36-891d-2ada964af0ea-kube-api-access-7ncq2\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:08.969938 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.969933 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:08.970064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:08.969944 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a977dd51-c671-4b36-891d-2ada964af0ea-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:09.689438 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.689410 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" Jul 10 13:17:09.689776 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.689448 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t" event={"ID":"a977dd51-c671-4b36-891d-2ada964af0ea","Type":"ContainerDied","Data":"1080a37fee692b033a6e85746e32519878d4b1fa9e8d894ba7115c1bf0904cb5"} Jul 10 13:17:09.689776 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.689518 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1080a37fee692b033a6e85746e32519878d4b1fa9e8d894ba7115c1bf0904cb5" Jul 10 13:17:09.814382 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.814353 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:09.876982 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.876959 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-bundle\") pod \"b2576174-c9b6-4b39-baac-05982b9e5f4e\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " Jul 10 13:17:09.877095 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.877047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-util\") pod \"b2576174-c9b6-4b39-baac-05982b9e5f4e\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " Jul 10 13:17:09.877095 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.877068 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf8bd\" (UniqueName: \"kubernetes.io/projected/b2576174-c9b6-4b39-baac-05982b9e5f4e-kube-api-access-zf8bd\") pod \"b2576174-c9b6-4b39-baac-05982b9e5f4e\" (UID: \"b2576174-c9b6-4b39-baac-05982b9e5f4e\") " Jul 10 13:17:09.877456 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.877429 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-bundle" (OuterVolumeSpecName: "bundle") pod "b2576174-c9b6-4b39-baac-05982b9e5f4e" (UID: "b2576174-c9b6-4b39-baac-05982b9e5f4e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:09.879208 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.879178 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2576174-c9b6-4b39-baac-05982b9e5f4e-kube-api-access-zf8bd" (OuterVolumeSpecName: "kube-api-access-zf8bd") pod "b2576174-c9b6-4b39-baac-05982b9e5f4e" (UID: "b2576174-c9b6-4b39-baac-05982b9e5f4e"). InnerVolumeSpecName "kube-api-access-zf8bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:17:09.882420 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.882397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-util" (OuterVolumeSpecName: "util") pod "b2576174-c9b6-4b39-baac-05982b9e5f4e" (UID: "b2576174-c9b6-4b39-baac-05982b9e5f4e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:17:09.978249 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.978226 2567 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-bundle\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:09.978249 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.978249 2567 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b2576174-c9b6-4b39-baac-05982b9e5f4e-util\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:09.978362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:09.978258 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf8bd\" (UniqueName: \"kubernetes.io/projected/b2576174-c9b6-4b39-baac-05982b9e5f4e-kube-api-access-zf8bd\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:17:10.695445 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:10.695419 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" Jul 10 13:17:10.695445 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:10.695435 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4" event={"ID":"b2576174-c9b6-4b39-baac-05982b9e5f4e","Type":"ContainerDied","Data":"c8372d934d572d177deaa646bc9c563213b687d8d31a31e9df668a767d87298c"} Jul 10 13:17:10.695887 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:10.695480 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8372d934d572d177deaa646bc9c563213b687d8d31a31e9df668a767d87298c" Jul 10 13:17:16.875271 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:16.875234 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:20.313084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:20.313045 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:21.902866 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:21.902819 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:22.340546 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:22.340511 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:22.918173 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:22.918136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:23.040576 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:23.040527 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:25.352943 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:25.352905 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:27.302295 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:27.302264 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:27.682952 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:27.682787 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:29.928866 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:29.928813 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:31.677612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:31.677581 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:31.755236 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:31.755208 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:31.924687 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:31.924655 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:32.169473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:32.169413 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:41.147557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:41.147516 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:41.312928 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:41.312839 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:41.439001 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:41.438902 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:43.398169 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:43.398139 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:43.499010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:43.498980 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:48.106046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:48.106016 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:48.214340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:48.214305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:51.919065 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:51.919035 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:17:52.036080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:17:52.036047 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:00.611004 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:00.610969 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:00.709166 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:00.709136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:01.269624 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:01.269593 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:01.371432 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:01.371401 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:03.052266 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:03.052231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:03.150381 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:03.150350 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:07.853995 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:07.853964 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:07.946380 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:07.946346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:11.806233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:11.806181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:11.917084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:11.917055 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:20.460868 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:20.460815 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:20.561695 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:20.561662 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:22.692484 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692453 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql"] Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692749 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="extract" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692759 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="extract" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692769 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="extract" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692774 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="extract" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692781 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="util" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692786 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="util" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692795 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="util" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692800 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="util" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692807 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="pull" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692812 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="pull" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692818 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="pull" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692823 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="pull" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692829 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="extract" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692834 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="extract" Jul 10 13:18:22.692860 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692871 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="util" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692878 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="util" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692886 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="extract" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692890 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="extract" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692898 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="pull" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692902 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="pull" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692911 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="pull" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692915 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="pull" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692924 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="util" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692928 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="util" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692978 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a4aba3-abee-4b36-bb31-5273afe623d0" containerName="extract" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692985 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a977dd51-c671-4b36-891d-2ada964af0ea" containerName="extract" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.692995 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2576174-c9b6-4b39-baac-05982b9e5f4e" containerName="extract" Jul 10 13:18:22.693331 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.693002 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="09b4ecc1-9b42-4c81-adc7-e378b8674627" containerName="extract" Jul 10 13:18:22.695974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.695956 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.698270 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.698251 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9slq4\"" Jul 10 13:18:22.711062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.711036 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql"] Jul 10 13:18:22.720049 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.720028 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:22.825120 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825085 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825141 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825161 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825186 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825208 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825225 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2cb\" (UniqueName: \"kubernetes.io/projected/236b4ab9-9245-43b0-bf7d-b58db9ac5895-kube-api-access-5d2cb\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825451 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825293 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825451 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825323 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.825451 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.825341 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.920326 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.920278 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:22.926401 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926376 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926531 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926409 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926531 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926432 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926642 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926534 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926642 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926559 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2cb\" (UniqueName: \"kubernetes.io/projected/236b4ab9-9245-43b0-bf7d-b58db9ac5895-kube-api-access-5d2cb\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926642 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926609 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926642 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926631 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926857 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926658 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926857 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926693 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.926970 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926832 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.927022 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.926960 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.927130 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.927108 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.927207 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.927132 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.927293 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.927269 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.929042 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.929019 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.929284 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.929268 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.936397 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.936371 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/236b4ab9-9245-43b0-bf7d-b58db9ac5895-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:22.937948 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:22.937924 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2cb\" (UniqueName: \"kubernetes.io/projected/236b4ab9-9245-43b0-bf7d-b58db9ac5895-kube-api-access-5d2cb\") pod \"maas-default-gateway-openshift-default-56654dcf7-g7mql\" (UID: \"236b4ab9-9245-43b0-bf7d-b58db9ac5895\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:23.005989 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:23.005964 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:23.020365 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:23.020341 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:23.144294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:23.144269 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql"] Jul 10 13:18:23.146202 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:18:23.146177 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236b4ab9_9245_43b0_bf7d_b58db9ac5895.slice/crio-9a89a95f49b5794443d74a0f1f3356864b53d69cb2a95a026bdb67703b4fdfc3 WatchSource:0}: Error finding container 9a89a95f49b5794443d74a0f1f3356864b53d69cb2a95a026bdb67703b4fdfc3: Status 404 returned error can't find the container with id 9a89a95f49b5794443d74a0f1f3356864b53d69cb2a95a026bdb67703b4fdfc3 Jul 10 13:18:23.148237 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:23.148207 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:18:23.148294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:23.148272 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:18:23.148329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:23.148298 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:18:24.043329 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:24.043288 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" event={"ID":"236b4ab9-9245-43b0-bf7d-b58db9ac5895","Type":"ContainerStarted","Data":"440e29baac7b461f0de05ee00c3891bc6fdc02de229b8fa1ddf098cde02b74a8"} Jul 10 13:18:24.043681 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:24.043340 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" event={"ID":"236b4ab9-9245-43b0-bf7d-b58db9ac5895","Type":"ContainerStarted","Data":"9a89a95f49b5794443d74a0f1f3356864b53d69cb2a95a026bdb67703b4fdfc3"} Jul 10 13:18:24.081374 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:24.081334 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" podStartSLOduration=2.081320677 podStartE2EDuration="2.081320677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:18:24.073217621 +0000 UTC m=+484.789595265" watchObservedRunningTime="2026-07-10 13:18:24.081320677 +0000 UTC m=+484.797698302" Jul 10 13:18:25.006697 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:25.006664 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:25.011445 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:25.011425 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:25.048482 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:25.048453 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:25.049325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:25.049309 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:18:25.500886 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:25.500811 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:25.790243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:25.790169 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:26.557090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:26.557062 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:26.731212 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:26.731181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:27.695499 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:27.695460 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:27.798691 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:27.798664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:31.737418 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:31.737381 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:46.419406 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.419375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:46.508198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.508166 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:46.889535 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.889505 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-p46pn"] Jul 10 13:18:46.892808 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.892786 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:46.896244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.896223 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 10 13:18:46.897145 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.897127 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-624f6\"" Jul 10 13:18:46.897244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.897161 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 13:18:46.897244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.897165 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 13:18:46.920732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.920711 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-p46pn"] Jul 10 13:18:46.985336 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:46.985317 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:47.014304 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.014281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkt7\" (UniqueName: \"kubernetes.io/projected/bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c-kube-api-access-5lkt7\") pod \"postgres-69fb4594cf-p46pn\" (UID: \"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.014413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.014356 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c-data\") pod \"postgres-69fb4594cf-p46pn\" (UID: \"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.114880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.114824 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkt7\" (UniqueName: \"kubernetes.io/projected/bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c-kube-api-access-5lkt7\") pod \"postgres-69fb4594cf-p46pn\" (UID: \"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.115058 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.114953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c-data\") pod \"postgres-69fb4594cf-p46pn\" (UID: \"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.115322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.115302 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c-data\") pod \"postgres-69fb4594cf-p46pn\" (UID: \"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.129112 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.129091 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkt7\" (UniqueName: \"kubernetes.io/projected/bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c-kube-api-access-5lkt7\") pod \"postgres-69fb4594cf-p46pn\" (UID: \"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.202560 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.202537 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:47.339536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:47.339508 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-p46pn"] Jul 10 13:18:47.341529 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:18:47.341503 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3a7c9c_c665_4802_bdf8_a4b1e11cdf3c.slice/crio-a1647b6a799904de232ade6ff9af8e934c12f293afc8b294a09ea82fce6685f8 WatchSource:0}: Error finding container a1647b6a799904de232ade6ff9af8e934c12f293afc8b294a09ea82fce6685f8: Status 404 returned error can't find the container with id a1647b6a799904de232ade6ff9af8e934c12f293afc8b294a09ea82fce6685f8 Jul 10 13:18:48.164520 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:48.164449 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" event={"ID":"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c","Type":"ContainerStarted","Data":"a1647b6a799904de232ade6ff9af8e934c12f293afc8b294a09ea82fce6685f8"} Jul 10 13:18:52.191596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:52.191561 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" event={"ID":"bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c","Type":"ContainerStarted","Data":"a036f816469a9acf8c50927b02c9c9014cb516507296612f9ef6c1ddd5a17d0d"} Jul 10 13:18:52.192016 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:52.191634 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:52.213425 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:52.213346 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" podStartSLOduration=1.601733909 podStartE2EDuration="6.213331929s" totalImagesPullingTime="4.61159802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:18:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:18:52.21127087 +0000 UTC m=+512.927648515" watchObservedRunningTime="2026-07-10 13:18:52.213331929 +0000 UTC m=+512.929709584" Jul 10 13:18:58.225971 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:58.225942 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-p46pn" Jul 10 13:18:58.527434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:58.527402 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:18:58.734382 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:18:58.734351 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:05.289094 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:05.289065 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:05.548315 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:05.548237 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:08.397365 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:08.397328 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:08.511343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:08.511309 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:24.649718 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:24.649645 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:24.749392 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:24.749361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:30.182568 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:30.182537 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:30.283392 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:30.283363 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:40.792294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:40.792257 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:41.372322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:41.372277 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:41.746060 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:41.746026 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:41.888136 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:41.888098 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:42.013320 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:42.013253 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:50.484246 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:50.484210 2567 generic.go:356] "Generic (PLEG): container finished" podID="236b4ab9-9245-43b0-bf7d-b58db9ac5895" containerID="440e29baac7b461f0de05ee00c3891bc6fdc02de229b8fa1ddf098cde02b74a8" exitCode=0 Jul 10 13:19:50.484246 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:50.484248 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" event={"ID":"236b4ab9-9245-43b0-bf7d-b58db9ac5895","Type":"ContainerDied","Data":"440e29baac7b461f0de05ee00c3891bc6fdc02de229b8fa1ddf098cde02b74a8"} Jul 10 13:19:50.484686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:50.484603 2567 scope.go:117] "RemoveContainer" containerID="440e29baac7b461f0de05ee00c3891bc6fdc02de229b8fa1ddf098cde02b74a8" Jul 10 13:19:50.485525 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:50.485499 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:19:50.485575 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:50.485564 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:19:50.485613 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:50.485592 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:19:51.490862 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:51.490814 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" event={"ID":"236b4ab9-9245-43b0-bf7d-b58db9ac5895","Type":"ContainerStarted","Data":"c9ee4ebeab534f4c7ce36b56498af55b2acc373057d75bff5c260c2de43c5c0d"} Jul 10 13:19:51.850403 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:51.850278 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:52.006612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:52.006577 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:19:52.007678 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:52.007657 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 10 13:19:52.007789 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:52.007709 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" podUID="236b4ab9-9245-43b0-bf7d-b58db9ac5895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 10 13:19:53.006550 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:53.006511 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 10 13:19:53.006933 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:53.006570 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" podUID="236b4ab9-9245-43b0-bf7d-b58db9ac5895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 10 13:19:53.547990 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:53.547955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:19:54.006936 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:54.006902 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 10 13:19:54.007318 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:54.006956 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" podUID="236b4ab9-9245-43b0-bf7d-b58db9ac5895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 10 13:19:55.007201 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:55.007169 2567 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-g7mql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 10 13:19:55.007554 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:55.007223 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" podUID="236b4ab9-9245-43b0-bf7d-b58db9ac5895" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 10 13:19:55.048719 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:55.048646 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:19:56.010271 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:56.010237 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:19:56.011211 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:19:56.011193 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-g7mql" Jul 10 13:20:01.543044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.543014 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh"] Jul 10 13:20:01.545822 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.545801 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.548224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.548204 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 13:20:01.549208 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.549190 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 13:20:01.549311 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.549292 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 13:20:01.549368 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.549283 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4jbww\"" Jul 10 13:20:01.558708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.558689 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh"] Jul 10 13:20:01.590059 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.590030 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:01.621267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.621245 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.621356 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.621287 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.621405 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.621352 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.621405 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.621380 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.621482 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.621409 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.621482 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.621448 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47x49\" (UniqueName: \"kubernetes.io/projected/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-kube-api-access-47x49\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.722627 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.722599 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47x49\" (UniqueName: \"kubernetes.io/projected/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-kube-api-access-47x49\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.722772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.722644 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.722772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.722687 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.722772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.722717 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.722772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.722744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.723015 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.722877 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.723201 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.723180 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.723240 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.723225 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.723281 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.723261 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.725103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.725087 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.725270 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.725254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.731029 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.731003 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47x49\" (UniqueName: \"kubernetes.io/projected/2e69cbcf-7d18-4237-b9cc-c1fc369a6171-kube-api-access-47x49\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh\" (UID: \"2e69cbcf-7d18-4237-b9cc-c1fc369a6171\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.855941 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.855873 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:01.993899 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:01.993877 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh"] Jul 10 13:20:01.995295 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:20:01.995271 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e69cbcf_7d18_4237_b9cc_c1fc369a6171.slice/crio-6b430b21758787eac295aeed6bd0304a9fe365689db1917d9b547f6755cbc115 WatchSource:0}: Error finding container 6b430b21758787eac295aeed6bd0304a9fe365689db1917d9b547f6755cbc115: Status 404 returned error can't find the container with id 6b430b21758787eac295aeed6bd0304a9fe365689db1917d9b547f6755cbc115 Jul 10 13:20:02.546504 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:02.546423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" event={"ID":"2e69cbcf-7d18-4237-b9cc-c1fc369a6171","Type":"ContainerStarted","Data":"6b430b21758787eac295aeed6bd0304a9fe365689db1917d9b547f6755cbc115"} Jul 10 13:20:03.552748 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:03.552662 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" event={"ID":"2e69cbcf-7d18-4237-b9cc-c1fc369a6171","Type":"ContainerStarted","Data":"83da7cf77164df00df7023affddfc3028bcc421c531cdf6f2881cf46fd61b785"} Jul 10 13:20:03.574888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:03.574825 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" podStartSLOduration=1.275753615 podStartE2EDuration="2.574810856s" totalImagesPullingTime="1.299057241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:20:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:20:03.572616067 +0000 UTC m=+584.288993723" watchObservedRunningTime="2026-07-10 13:20:03.574810856 +0000 UTC m=+584.291188554" Jul 10 13:20:11.856104 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:11.856055 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:11.856462 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:11.856133 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:11.870888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:11.870856 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:12.607252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:12.607218 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh" Jul 10 13:20:15.294339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:15.294300 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:15.437465 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:15.437437 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:15.633902 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:15.633781 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:19.777377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:19.777347 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:20:19.779390 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:19.779368 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:20:21.113374 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.113337 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m"] Jul 10 13:20:21.118248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.118213 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.131505 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.131486 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 13:20:21.154584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.154562 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m"] Jul 10 13:20:21.174128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.174107 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.174235 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.174144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.174235 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.174187 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtls\" (UniqueName: \"kubernetes.io/projected/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-kube-api-access-nwtls\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.174309 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.174252 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.174309 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.174284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.174309 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.174300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.175917 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.175896 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:21.275224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275195 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275243 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtls\" (UniqueName: \"kubernetes.io/projected/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-kube-api-access-nwtls\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275377 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275509 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275387 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275509 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275422 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275721 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275693 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275721 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.275824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.275800 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.277597 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.277578 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.277694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.277677 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.285002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.284978 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtls\" (UniqueName: \"kubernetes.io/projected/4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2-kube-api-access-nwtls\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5ct8m\" (UID: \"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.428233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.428175 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:21.576773 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.576750 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m"] Jul 10 13:20:21.579385 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:20:21.579352 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc6ce88_6e33_4f1c_ae1b_35fa341da8a2.slice/crio-f31f5a71073b2b785d85b58e69e5a8031cf2e7187b862170a0d5bad0b8e169bb WatchSource:0}: Error finding container f31f5a71073b2b785d85b58e69e5a8031cf2e7187b862170a0d5bad0b8e169bb: Status 404 returned error can't find the container with id f31f5a71073b2b785d85b58e69e5a8031cf2e7187b862170a0d5bad0b8e169bb Jul 10 13:20:21.642502 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:21.642475 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" event={"ID":"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2","Type":"ContainerStarted","Data":"f31f5a71073b2b785d85b58e69e5a8031cf2e7187b862170a0d5bad0b8e169bb"} Jul 10 13:20:22.649327 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:22.649290 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" event={"ID":"4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2","Type":"ContainerStarted","Data":"a2c8c20443a0f80aa7d4176675f55c5ef235fba777d4d04f01533ca04e8be14b"} Jul 10 13:20:22.673506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:22.673460 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" podStartSLOduration=1.434343514 podStartE2EDuration="1.673445578s" totalImagesPullingTime="239.102064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:20:22.671233168 +0000 UTC m=+603.387610795" watchObservedRunningTime="2026-07-10 13:20:22.673445578 +0000 UTC m=+603.389823221" Jul 10 13:20:31.428733 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:31.428696 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:31.429224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:31.428750 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:31.440832 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:31.440810 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:31.704989 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:31.704964 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5ct8m" Jul 10 13:20:34.508502 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:34.508474 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:52.752313 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:52.752233 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:20:52.815681 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:20:52.815649 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:21:04.836614 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:21:04.836578 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:21:08.709834 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:21:08.709801 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:21:58.031915 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:21:58.031875 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:21:58.132822 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:21:58.132784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:21:59.376626 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:21:59.376596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:21:59.477647 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:21:59.477616 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:22:19.728302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:22:19.728221 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:22:35.342065 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:22:35.342027 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:22:57.317746 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:22:57.317713 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:23:13.220928 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:23:13.220885 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:23:14.477404 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:23:14.477367 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:23:55.728322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:23:55.728241 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:23:55.826466 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:23:55.826435 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:23:55.963003 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:23:55.962973 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:01.732395 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:01.732362 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:01.802397 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:01.802372 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:01.902950 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:01.902923 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:02.001390 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:02.001321 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:02.101318 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:02.101293 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:07.710779 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:07.710744 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:24:07.869141 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:24:07.869112 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:25:19.567217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:25:19.567136 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:25:19.809418 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:25:19.809374 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:25:19.812916 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:25:19.812897 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:25:21.180902 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:25:21.180872 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:27:35.119415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:27:35.119371 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:29:00.655870 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:29:00.655813 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:30:08.733745 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:30:08.733704 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:30:19.841396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:30:19.841364 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:30:19.849167 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:30:19.849141 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:35:08.037672 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:08.037626 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:09.464399 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:09.464361 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:13.757488 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.757451 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:13.828536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.828497 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:13.953644 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.953619 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq"] Jul 10 13:35:13.956804 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.956787 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:13.959433 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.959413 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2a72c3fa-openshift-default-dockercfg-vkdcp\"" Jul 10 13:35:13.991354 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.991331 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq"] Jul 10 13:35:13.993993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:13.993971 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:14.077892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.077804 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-envoy\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.077892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.077837 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-podinfo\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.077892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.077871 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istiod-ca-cert\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.078098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.077904 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-data\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.078098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.077966 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-token\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.078098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.077985 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn7qn\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-kube-api-access-xn7qn\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.078098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.078014 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-certs\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.078098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.078039 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-socket\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.078098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.078061 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-credential-socket\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.093484 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.093454 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:14.179007 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.178982 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-certs\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179010 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-socket\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179030 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-credential-socket\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179060 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-envoy\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179119 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179097 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-podinfo\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179284 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179125 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istiod-ca-cert\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179284 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179152 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-data\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179284 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179232 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-token\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179284 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179256 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn7qn\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-kube-api-access-xn7qn\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179460 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179436 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-certs\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179575 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-credential-socket\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179831 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179806 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-data\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179944 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179895 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istiod-ca-cert\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.179944 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.179896 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-socket\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.181422 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.181406 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-envoy\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.181500 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.181483 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-podinfo\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.192172 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.192148 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn7qn\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-kube-api-access-xn7qn\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.194878 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.194861 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-token\") pod \"e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.266773 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.266739 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:14.413142 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.413121 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq"] Jul 10 13:35:14.414964 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:35:14.414935 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc2393c_b916_49cc_bb0f_cdb8207b1836.slice/crio-5a33dbf9d9a74ee5eb85a9e669bd914fde7399871235ef63033328b8860406c1 WatchSource:0}: Error finding container 5a33dbf9d9a74ee5eb85a9e669bd914fde7399871235ef63033328b8860406c1: Status 404 returned error can't find the container with id 5a33dbf9d9a74ee5eb85a9e669bd914fde7399871235ef63033328b8860406c1 Jul 10 13:35:14.417038 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.417007 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:14.417111 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.417074 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:14.417111 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:14.417104 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:15.015824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.015792 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" event={"ID":"4fc2393c-b916-49cc-bb0f-cdb8207b1836","Type":"ContainerStarted","Data":"0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2"} Jul 10 13:35:15.016221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.015865 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" event={"ID":"4fc2393c-b916-49cc-bb0f-cdb8207b1836","Type":"ContainerStarted","Data":"5a33dbf9d9a74ee5eb85a9e669bd914fde7399871235ef63033328b8860406c1"} Jul 10 13:35:15.041304 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.041262 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" podStartSLOduration=2.041246126 podStartE2EDuration="2.041246126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:35:15.038717111 +0000 UTC m=+1495.755094737" watchObservedRunningTime="2026-07-10 13:35:15.041246126 +0000 UTC m=+1495.757623752" Jul 10 13:35:15.183665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.183628 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:15.267469 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.267397 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:15.268984 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.268944 2567 patch_prober.go:28] interesting pod/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Jul 10 13:35:15.269152 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.269012 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" podUID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Jul 10 13:35:15.303262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.303232 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:15.407393 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.407345 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:15.755958 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:15.755924 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:16.023186 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:16.023095 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:16.118438 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:16.118408 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:16.270786 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:16.270753 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:17.026268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:17.026239 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:17.027293 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:17.027268 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:17.831401 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:17.831369 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:19.877372 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:19.877344 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:35:19.888122 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:19.888101 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:35:20.271332 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:20.271303 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:20.431817 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:20.431778 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq"] Jul 10 13:35:20.432085 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:20.432034 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" podUID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" containerName="istio-proxy" containerID="cri-o://0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2" gracePeriod=30 Jul 10 13:35:20.457317 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:20.457289 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:20.560274 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:20.560196 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:20.681762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:20.681728 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:21.757540 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:21.757509 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:26.278793 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.278745 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:26.375162 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.375125 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:26.380438 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.380408 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:26.447744 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.447715 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv"] Jul 10 13:35:26.448230 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.448212 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" containerName="istio-proxy" Jul 10 13:35:26.448230 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.448233 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" containerName="istio-proxy" Jul 10 13:35:26.448354 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.448331 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" containerName="istio-proxy" Jul 10 13:35:26.450657 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.450630 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.453975 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.453953 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-28b6c6a0-openshift-default-dockercfg-nkh2g\"" Jul 10 13:35:26.467103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.466617 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv"] Jul 10 13:35:26.499812 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.499784 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:26.512411 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512391 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-certs\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512517 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512437 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-data\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512517 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512461 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istiod-ca-cert\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512517 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512477 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xn7qn\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-kube-api-access-xn7qn\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512517 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512501 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-socket\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512560 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-token\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512611 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-podinfo\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512652 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-credential-socket\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512680 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-envoy\") pod \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\" (UID: \"4fc2393c-b916-49cc-bb0f-cdb8207b1836\") " Jul 10 13:35:26.512716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512705 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:26.513002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512801 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:26.513002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.512979 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:26.513135 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.513108 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.513206 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.513135 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.513206 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.513156 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.513308 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.513210 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-data" (OuterVolumeSpecName: "istio-data") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:26.513308 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.513261 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:35:26.514839 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.514811 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-kube-api-access-xn7qn" (OuterVolumeSpecName: "kube-api-access-xn7qn") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "kube-api-access-xn7qn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:26.515000 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.514968 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:35:26.515263 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.515243 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:26.515351 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.515326 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-token" (OuterVolumeSpecName: "istio-token") pod "4fc2393c-b916-49cc-bb0f-cdb8207b1836" (UID: "4fc2393c-b916-49cc-bb0f-cdb8207b1836"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:26.600252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.600214 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:26.613966 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.613942 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-socket\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614069 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.613974 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbbcc22-076a-400e-8514-5fe893d6a171-istiod-ca-cert\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614080 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-podinfo\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614120 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-token\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614223 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614148 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vj88\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-kube-api-access-9vj88\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614217 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-certs\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614285 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-data\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614312 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-credential-socket\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614331 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-envoy\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.614411 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614390 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.614411 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614402 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.614483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614412 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn7qn\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-kube-api-access-xn7qn\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.614483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614420 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.614483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614428 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.614483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.614436 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fc2393c-b916-49cc-bb0f-cdb8207b1836-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:26.715255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715189 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-socket\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715214 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbbcc22-076a-400e-8514-5fe893d6a171-istiod-ca-cert\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715258 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-podinfo\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715301 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-token\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715331 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vj88\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-kube-api-access-9vj88\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715379 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-certs\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715426 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-data\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715456 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-credential-socket\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715485 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-envoy\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715781 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715730 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-socket\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.715821 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715804 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-certs\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.716011 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.715987 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbbcc22-076a-400e-8514-5fe893d6a171-istiod-ca-cert\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.716078 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.716050 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-data\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.716136 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.716103 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-credential-socket\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.717690 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.717672 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-envoy\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.717935 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.717917 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-podinfo\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.723219 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.723197 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-token\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.723380 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.723364 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vj88\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-kube-api-access-9vj88\") pod \"e2e-ait-28b6c6a0-openshift-default-b64774859-986rv\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.768480 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.768459 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:26.899292 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.899266 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv"] Jul 10 13:35:26.900129 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:35:26.900105 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbbcc22_076a_400e_8514_5fe893d6a171.slice/crio-6033f1e7fa8b2969d9073ed4284ec82c7f038ad71596d87a0a5de8b58714615e WatchSource:0}: Error finding container 6033f1e7fa8b2969d9073ed4284ec82c7f038ad71596d87a0a5de8b58714615e: Status 404 returned error can't find the container with id 6033f1e7fa8b2969d9073ed4284ec82c7f038ad71596d87a0a5de8b58714615e Jul 10 13:35:26.902241 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.902209 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:26.902325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.902274 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:26.902325 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:26.902307 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:27.097869 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.097747 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" event={"ID":"3dbbcc22-076a-400e-8514-5fe893d6a171","Type":"ContainerStarted","Data":"92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94"} Jul 10 13:35:27.098037 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.097875 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" event={"ID":"3dbbcc22-076a-400e-8514-5fe893d6a171","Type":"ContainerStarted","Data":"6033f1e7fa8b2969d9073ed4284ec82c7f038ad71596d87a0a5de8b58714615e"} Jul 10 13:35:27.099234 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.099208 2567 generic.go:356] "Generic (PLEG): container finished" podID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" containerID="0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2" exitCode=0 Jul 10 13:35:27.099379 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.099239 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" event={"ID":"4fc2393c-b916-49cc-bb0f-cdb8207b1836","Type":"ContainerDied","Data":"0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2"} Jul 10 13:35:27.099379 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.099252 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" Jul 10 13:35:27.099379 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.099269 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq" event={"ID":"4fc2393c-b916-49cc-bb0f-cdb8207b1836","Type":"ContainerDied","Data":"5a33dbf9d9a74ee5eb85a9e669bd914fde7399871235ef63033328b8860406c1"} Jul 10 13:35:27.099379 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.099295 2567 scope.go:117] "RemoveContainer" containerID="0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2" Jul 10 13:35:27.108824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.108804 2567 scope.go:117] "RemoveContainer" containerID="0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2" Jul 10 13:35:27.109077 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:35:27.109049 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2\": container with ID starting with 0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2 not found: ID does not exist" containerID="0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2" Jul 10 13:35:27.109154 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.109087 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2"} err="failed to get container status \"0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2\": rpc error: code = NotFound desc = could not find container \"0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2\": container with ID starting with 0dfa5fa9b2d03bf99261a2c64187065a82db35851ec9cc8ef8e25e88fcbd4dd2 not found: ID does not exist" Jul 10 13:35:27.146126 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.146082 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" podStartSLOduration=1.1460671119999999 podStartE2EDuration="1.146067112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:35:27.143408308 +0000 UTC m=+1507.859785924" watchObservedRunningTime="2026-07-10 13:35:27.146067112 +0000 UTC m=+1507.862444816" Jul 10 13:35:27.171265 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.171243 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq"] Jul 10 13:35:27.195764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.195739 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2a72c3fa-openshift-default-58d6cbfc-7rncq"] Jul 10 13:35:27.666629 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.666591 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:27.766061 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.766007 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:27.769169 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.769145 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:27.773538 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.773520 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:27.866770 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.866727 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:27.951565 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.951539 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fc2393c-b916-49cc-bb0f-cdb8207b1836" path="/var/lib/kubelet/pods/4fc2393c-b916-49cc-bb0f-cdb8207b1836/volumes" Jul 10 13:35:27.964412 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:27.964386 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:28.105076 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:28.105052 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:28.105877 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:28.105839 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:28.348903 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:28.348788 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:28.444652 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:28.444619 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:29.962256 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:29.962231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:32.708976 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:32.708923 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:33.092666 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:33.092588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:37.154159 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:37.154135 2567 generic.go:356] "Generic (PLEG): container finished" podID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerID="92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94" exitCode=0 Jul 10 13:35:37.154485 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:37.154209 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" event={"ID":"3dbbcc22-076a-400e-8514-5fe893d6a171","Type":"ContainerDied","Data":"92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94"} Jul 10 13:35:37.154624 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:37.154608 2567 scope.go:117] "RemoveContainer" containerID="92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94" Jul 10 13:35:37.155562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:37.155539 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:37.155628 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:37.155607 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:37.155664 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:37.155638 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:38.161567 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:38.161537 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" event={"ID":"3dbbcc22-076a-400e-8514-5fe893d6a171","Type":"ContainerStarted","Data":"0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548"} Jul 10 13:35:38.769295 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:38.769259 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:38.774146 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:38.774119 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:38.858819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:38.858788 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv"] Jul 10 13:35:38.883024 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:38.882989 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:38.998622 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:38.998583 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:39.167473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:39.167396 2567 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" secret="" err="secret \"e2e-ait-28b6c6a0-openshift-default-dockercfg-nkh2g\" not found" Jul 10 13:35:39.167473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:39.167441 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:39.168515 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:39.168498 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:40.172339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:40.172303 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" containerID="cri-o://0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548" gracePeriod=30 Jul 10 13:35:41.645270 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:41.645232 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:44.867721 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:44.867670 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:44.933362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:44.933326 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:45.082575 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.082542 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd"] Jul 10 13:35:45.085384 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.085362 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.088171 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.088152 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3763cabf-openshift-default-dockercfg-rx9kr\"" Jul 10 13:35:45.105547 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.105526 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd"] Jul 10 13:35:45.153539 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.153473 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:45.176277 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176253 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-socket\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176309 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-envoy\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176344 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-podinfo\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176375 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-token\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istiod-ca-cert\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176431 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-credential-socket\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176453 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-data\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176498 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhff\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-kube-api-access-wdhff\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.176668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.176535 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-certs\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.224437 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.224403 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:45.277965 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.277942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhff\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-kube-api-access-wdhff\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.277984 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-certs\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278005 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-socket\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-envoy\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278092 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-podinfo\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278123 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-token\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278172 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istiod-ca-cert\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278201 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-credential-socket\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278229 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-data\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278619 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278519 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-socket\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278769 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-credential-socket\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278941 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278867 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-certs\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.278941 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.278912 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-data\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.279239 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.279221 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istiod-ca-cert\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.280776 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.280753 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-envoy\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.280906 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.280884 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-podinfo\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.288188 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.288157 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhff\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-kube-api-access-wdhff\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.288904 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.288885 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-token\") pod \"e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.397557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.397532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:45.409361 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.409343 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:45.480130 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480109 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-certs\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480164 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-data\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480189 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbbcc22-076a-400e-8514-5fe893d6a171-istiod-ca-cert\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480209 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-socket\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480380 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480238 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-token\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480380 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480263 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vj88\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-kube-api-access-9vj88\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480380 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480287 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-envoy\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480380 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480347 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-credential-socket\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480588 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480380 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-podinfo\") pod \"3dbbcc22-076a-400e-8514-5fe893d6a171\" (UID: \"3dbbcc22-076a-400e-8514-5fe893d6a171\") " Jul 10 13:35:45.480643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480607 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbbcc22-076a-400e-8514-5fe893d6a171-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:35:45.480938 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480875 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:45.480938 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480895 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:45.480938 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.480904 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:45.481141 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.481010 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-data" (OuterVolumeSpecName: "istio-data") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:45.483080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.483036 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:45.483275 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.483250 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-token" (OuterVolumeSpecName: "istio-token") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:45.483350 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.483272 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:35:45.483713 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.483688 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-kube-api-access-9vj88" (OuterVolumeSpecName: "kube-api-access-9vj88") pod "3dbbcc22-076a-400e-8514-5fe893d6a171" (UID: "3dbbcc22-076a-400e-8514-5fe893d6a171"). InnerVolumeSpecName "kube-api-access-9vj88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:45.534724 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.534698 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd"] Jul 10 13:35:45.536736 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:35:45.536710 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb2ed6e_ddad_4e24_a27a_a86d9a3a2e5d.slice/crio-ce71fe4709a21e12e5bbd2211f28ad18fe9589d9610d676e75aff91e56e1dfc8 WatchSource:0}: Error finding container ce71fe4709a21e12e5bbd2211f28ad18fe9589d9610d676e75aff91e56e1dfc8: Status 404 returned error can't find the container with id ce71fe4709a21e12e5bbd2211f28ad18fe9589d9610d676e75aff91e56e1dfc8 Jul 10 13:35:45.538798 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.538771 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:45.538888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.538831 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:45.538933 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.538893 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:35:45.581455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581434 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581456 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581469 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dbbcc22-076a-400e-8514-5fe893d6a171-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581481 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581492 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581505 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vj88\" (UniqueName: \"kubernetes.io/projected/3dbbcc22-076a-400e-8514-5fe893d6a171-kube-api-access-9vj88\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581517 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581530 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dbbcc22-076a-400e-8514-5fe893d6a171-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:45.581592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:45.581542 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dbbcc22-076a-400e-8514-5fe893d6a171-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:46.205444 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.205414 2567 generic.go:356] "Generic (PLEG): container finished" podID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerID="0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548" exitCode=0 Jul 10 13:35:46.205786 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.205461 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" Jul 10 13:35:46.205786 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.205486 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" event={"ID":"3dbbcc22-076a-400e-8514-5fe893d6a171","Type":"ContainerDied","Data":"0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548"} Jul 10 13:35:46.205786 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.205544 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv" event={"ID":"3dbbcc22-076a-400e-8514-5fe893d6a171","Type":"ContainerDied","Data":"6033f1e7fa8b2969d9073ed4284ec82c7f038ad71596d87a0a5de8b58714615e"} Jul 10 13:35:46.205983 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.205579 2567 scope.go:117] "RemoveContainer" containerID="0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548" Jul 10 13:35:46.211508 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.211306 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" event={"ID":"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d","Type":"ContainerStarted","Data":"5f4a29f322a195f5375eb10f5c2d9e775356adc82ff3960e6ac8c20012d13949"} Jul 10 13:35:46.211508 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.211393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" event={"ID":"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d","Type":"ContainerStarted","Data":"ce71fe4709a21e12e5bbd2211f28ad18fe9589d9610d676e75aff91e56e1dfc8"} Jul 10 13:35:46.220005 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.219985 2567 scope.go:117] "RemoveContainer" containerID="92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94" Jul 10 13:35:46.227596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.227580 2567 scope.go:117] "RemoveContainer" containerID="0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548" Jul 10 13:35:46.227861 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:35:46.227824 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548\": container with ID starting with 0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548 not found: ID does not exist" containerID="0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548" Jul 10 13:35:46.227909 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.227860 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548"} err="failed to get container status \"0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548\": rpc error: code = NotFound desc = could not find container \"0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548\": container with ID starting with 0e1875dcd3ad1d98e1b8ccb514d9cf02e9f1150c26e85f8096707cfe5ef2a548 not found: ID does not exist" Jul 10 13:35:46.227909 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.227875 2567 scope.go:117] "RemoveContainer" containerID="92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94" Jul 10 13:35:46.228081 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:35:46.228065 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94\": container with ID starting with 92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94 not found: ID does not exist" containerID="92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94" Jul 10 13:35:46.228126 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.228085 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94"} err="failed to get container status \"92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94\": rpc error: code = NotFound desc = could not find container \"92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94\": container with ID starting with 92ef2fabf0df7f9fe728af0366831018a27f02c20405d840c761b2679e27ec94 not found: ID does not exist" Jul 10 13:35:46.233159 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.233112 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" podStartSLOduration=1.233097129 podStartE2EDuration="1.233097129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:35:46.231367679 +0000 UTC m=+1526.947745314" watchObservedRunningTime="2026-07-10 13:35:46.233097129 +0000 UTC m=+1526.949474758" Jul 10 13:35:46.263937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.263914 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv"] Jul 10 13:35:46.268828 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.268802 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-28b6c6a0-openshift-default-b64774859-986rv"] Jul 10 13:35:46.323693 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.323666 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:46.398704 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.398676 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:46.403055 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.403035 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:46.477236 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.477155 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:46.596965 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.596934 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:46.785981 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:46.785875 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:47.090350 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:47.090247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:47.192640 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:47.192605 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:47.217317 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:47.217291 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:47.218208 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:47.218187 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:47.951117 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:47.951082 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" path="/var/lib/kubelet/pods/3dbbcc22-076a-400e-8514-5fe893d6a171/volumes" Jul 10 13:35:48.755580 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:48.755544 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:50.659916 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:50.659878 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:50.861986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:50.861911 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd"] Jul 10 13:35:50.862652 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:50.862621 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" podUID="deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" containerName="istio-proxy" containerID="cri-o://5f4a29f322a195f5375eb10f5c2d9e775356adc82ff3960e6ac8c20012d13949" gracePeriod=30 Jul 10 13:35:50.863372 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:50.863030 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:50.960948 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:50.960906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:51.022693 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:51.022652 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:52.499986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:52.499956 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:56.757416 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:56.757376 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:56.824623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:56.824576 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:57.280305 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.280279 2567 generic.go:356] "Generic (PLEG): container finished" podID="deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" containerID="5f4a29f322a195f5375eb10f5c2d9e775356adc82ff3960e6ac8c20012d13949" exitCode=0 Jul 10 13:35:57.280425 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.280352 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" event={"ID":"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d","Type":"ContainerDied","Data":"5f4a29f322a195f5375eb10f5c2d9e775356adc82ff3960e6ac8c20012d13949"} Jul 10 13:35:57.404312 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.404292 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:57.486562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486494 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-podinfo\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.486708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486561 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-token\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.486708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486587 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istiod-ca-cert\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.486708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486629 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-data\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.486708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486644 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-certs\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.486708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486698 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-socket\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.486996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486729 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-envoy\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.487064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.486997 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:57.487064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487024 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:35:57.487188 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487105 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdhff\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-kube-api-access-wdhff\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.487188 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487130 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:57.487188 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487165 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-credential-socket\") pod \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\" (UID: \"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d\") " Jul 10 13:35:57.487500 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487475 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.487500 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487500 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.487652 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487515 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.487706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487657 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:57.487942 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.487905 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-data" (OuterVolumeSpecName: "istio-data") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:57.488937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.488912 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-token" (OuterVolumeSpecName: "istio-token") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:57.489039 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.488944 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:35:57.489231 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.489204 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-kube-api-access-wdhff" (OuterVolumeSpecName: "kube-api-access-wdhff") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "kube-api-access-wdhff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:35:57.489373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.489348 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" (UID: "deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:35:57.588091 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.588063 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.588091 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.588089 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.588261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.588099 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.588261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.588107 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.588261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.588115 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.588261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.588123 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhff\" (UniqueName: \"kubernetes.io/projected/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d-kube-api-access-wdhff\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:35:57.622740 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.622714 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:57.719076 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.719026 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:57.819014 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:57.818948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:58.286694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:58.286672 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" Jul 10 13:35:58.286694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:58.286684 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd" event={"ID":"deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d","Type":"ContainerDied","Data":"ce71fe4709a21e12e5bbd2211f28ad18fe9589d9610d676e75aff91e56e1dfc8"} Jul 10 13:35:58.286929 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:58.286737 2567 scope.go:117] "RemoveContainer" containerID="5f4a29f322a195f5375eb10f5c2d9e775356adc82ff3960e6ac8c20012d13949" Jul 10 13:35:58.306984 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:58.306963 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd"] Jul 10 13:35:58.311499 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:58.311479 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3763cabf-openshift-default-9f6988b59-rs5jd"] Jul 10 13:35:58.717258 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:58.717223 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:59.227926 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:59.227895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:35:59.951626 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:35:59.951595 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" path="/var/lib/kubelet/pods/deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d/volumes" Jul 10 13:36:05.222892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:05.222863 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:05.324820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:05.324787 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:05.418744 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:05.418710 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:05.518755 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:05.518673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:06.008600 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:06.008568 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:31.014599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:31.014563 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:31.943246 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:31.943204 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:37.790008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:37.789971 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:37.972562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:37.972532 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:38.073528 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:38.073458 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:53.873600 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:53.873555 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:59.595638 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:59.595591 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:59.699774 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:59.699733 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:36:59.798783 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:36:59.798750 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:10.313262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:10.313230 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:11.087979 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:11.087948 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:16.908603 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:16.908566 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:16.975098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:16.975059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:17.415757 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:17.415716 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:17.515671 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:17.515631 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:17.615385 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:17.615332 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:18.382117 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:18.382072 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:18.512295 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:18.512245 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:18.632518 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:18.632434 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:20.338359 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:20.338320 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:26.083184 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:26.083143 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:26.144024 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:26.143992 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:26.778189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:26.778155 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:32.777935 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:32.777900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:32.842360 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:32.842329 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:33.041349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.041271 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:33.359396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359310 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr"] Jul 10 13:37:33.359661 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359648 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" Jul 10 13:37:33.359703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359662 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" Jul 10 13:37:33.359703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359675 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" Jul 10 13:37:33.359703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359680 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" Jul 10 13:37:33.359703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359686 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" containerName="istio-proxy" Jul 10 13:37:33.359703 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359692 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" containerName="istio-proxy" Jul 10 13:37:33.359870 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359747 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" Jul 10 13:37:33.359870 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.359757 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb2ed6e-ddad-4e24-a27a-a86d9a3a2e5d" containerName="istio-proxy" Jul 10 13:37:33.361894 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.361877 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.364415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.364397 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-69fdb29c-openshift-default-dockercfg-vbgms\"" Jul 10 13:37:33.386639 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.386611 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr"] Jul 10 13:37:33.390658 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.390612 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:33.455210 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455172 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-token\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455210 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455212 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-credential-socket\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455443 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455251 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-certs\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455443 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455280 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-envoy\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455443 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455364 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729f15e-1443-40be-b0fc-4d9c3ae46844-istiod-ca-cert\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455480 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmjx\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-kube-api-access-vxmjx\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455507 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-podinfo\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455528 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-data\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.455706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.455593 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-socket\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.556895 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.556863 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-data\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.556915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-socket\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.556963 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-token\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.556989 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-credential-socket\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557031 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-certs\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557061 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-envoy\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557332 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557084 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729f15e-1443-40be-b0fc-4d9c3ae46844-istiod-ca-cert\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557332 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557144 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmjx\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-kube-api-access-vxmjx\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557332 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557173 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-podinfo\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557332 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557272 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-data\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557527 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557425 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-socket\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557527 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557465 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-certs\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557560 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-credential-socket\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.557791 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.557771 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729f15e-1443-40be-b0fc-4d9c3ae46844-istiod-ca-cert\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.559275 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.559256 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-envoy\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.559524 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.559508 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-podinfo\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.565242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.565220 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-token\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.565471 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.565449 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmjx\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-kube-api-access-vxmjx\") pod \"e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.672478 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.672424 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:33.750600 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.750563 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:33.809356 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.809331 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr"] Jul 10 13:37:33.811435 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:37:33.811401 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd729f15e_1443_40be_b0fc_4d9c3ae46844.slice/crio-2a55f81cd55459e13891367a6c6e225dc32aa025ed934c1837a8b848ef1a9d7f WatchSource:0}: Error finding container 2a55f81cd55459e13891367a6c6e225dc32aa025ed934c1837a8b848ef1a9d7f: Status 404 returned error can't find the container with id 2a55f81cd55459e13891367a6c6e225dc32aa025ed934c1837a8b848ef1a9d7f Jul 10 13:37:33.813700 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.813671 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:37:33.813769 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.813734 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:37:33.813769 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:33.813763 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:37:34.760332 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:34.760296 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" event={"ID":"d729f15e-1443-40be-b0fc-4d9c3ae46844","Type":"ContainerStarted","Data":"4afe3c1f8eccd9199c90ad92871ee6541c32d2e399bb150679fdfb510d740554"} Jul 10 13:37:34.760467 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:34.760356 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" event={"ID":"d729f15e-1443-40be-b0fc-4d9c3ae46844","Type":"ContainerStarted","Data":"2a55f81cd55459e13891367a6c6e225dc32aa025ed934c1837a8b848ef1a9d7f"} Jul 10 13:37:34.784128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:34.784073 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" podStartSLOduration=1.784058771 podStartE2EDuration="1.784058771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:37:34.781040387 +0000 UTC m=+1635.497418013" watchObservedRunningTime="2026-07-10 13:37:34.784058771 +0000 UTC m=+1635.500436396" Jul 10 13:37:35.673453 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:35.673414 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:35.678532 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:35.678506 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:35.765855 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:35.765825 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:35.766733 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:35.766717 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:36.737490 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:36.737450 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:36.840924 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:36.840894 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:36.912566 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:36.912536 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:37.034278 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:37.034197 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:37.357679 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:37.357592 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:37.458579 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:37.458551 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:41.311423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:41.311384 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:42.067546 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:42.067517 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:42.182012 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:42.181967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:42.497002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:42.496970 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:48.151535 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:48.151502 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:48.250476 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:48.250341 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:48.735684 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:48.735646 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:49.089525 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:49.089439 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:49.189794 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:49.189752 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:51.803185 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.802776 2567 patch_prober.go:28] interesting pod/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 13:37:51.803185 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.802874 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 13:37:51.851319 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.851297 2567 generic.go:356] "Generic (PLEG): container finished" podID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerID="4afe3c1f8eccd9199c90ad92871ee6541c32d2e399bb150679fdfb510d740554" exitCode=0 Jul 10 13:37:51.851427 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.851372 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" event={"ID":"d729f15e-1443-40be-b0fc-4d9c3ae46844","Type":"ContainerDied","Data":"4afe3c1f8eccd9199c90ad92871ee6541c32d2e399bb150679fdfb510d740554"} Jul 10 13:37:51.851730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.851717 2567 scope.go:117] "RemoveContainer" containerID="4afe3c1f8eccd9199c90ad92871ee6541c32d2e399bb150679fdfb510d740554" Jul 10 13:37:51.852677 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.852653 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:37:51.852772 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.852736 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:37:51.852833 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:51.852807 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:37:52.859742 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:52.859704 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" event={"ID":"d729f15e-1443-40be-b0fc-4d9c3ae46844","Type":"ContainerStarted","Data":"2fccaeee418818c47bbf99093e439c3a1d981fa67de9f6aca2aaeae78e1db670"} Jul 10 13:37:53.672858 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:53.672798 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:37:53.673049 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:53.673028 2567 patch_prober.go:28] interesting pod/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 10 13:37:53.673158 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:53.673084 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 10 13:37:54.673473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:54.673440 2567 patch_prober.go:28] interesting pod/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Jul 10 13:37:54.673834 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:54.673498 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Jul 10 13:37:54.854892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:54.854830 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:54.921336 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:54.921307 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:55.131113 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.131078 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr"] Jul 10 13:37:55.140109 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.140072 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" containerID="cri-o://2fccaeee418818c47bbf99093e439c3a1d981fa67de9f6aca2aaeae78e1db670" gracePeriod=30 Jul 10 13:37:55.155205 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.155180 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:55.374933 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.374895 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:55.505676 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.505644 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:55.600908 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.600871 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:37:55.722256 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:37:55.722227 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:01.461010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.460972 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:01.531227 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.531200 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:01.625810 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.625779 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:01.726976 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.726914 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:01.825836 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.825806 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:01.907560 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.907540 2567 generic.go:356] "Generic (PLEG): container finished" podID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerID="2fccaeee418818c47bbf99093e439c3a1d981fa67de9f6aca2aaeae78e1db670" exitCode=0 Jul 10 13:38:01.907672 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.907608 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" event={"ID":"d729f15e-1443-40be-b0fc-4d9c3ae46844","Type":"ContainerDied","Data":"2fccaeee418818c47bbf99093e439c3a1d981fa67de9f6aca2aaeae78e1db670"} Jul 10 13:38:01.907672 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:01.907663 2567 scope.go:117] "RemoveContainer" containerID="4afe3c1f8eccd9199c90ad92871ee6541c32d2e399bb150679fdfb510d740554" Jul 10 13:38:02.292511 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.292492 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:38:02.414712 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414633 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-envoy\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.414712 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414667 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-certs\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.414712 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414697 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxmjx\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-kube-api-access-vxmjx\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414730 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-token\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414800 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729f15e-1443-40be-b0fc-4d9c3ae46844-istiod-ca-cert\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414835 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-credential-socket\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414871 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-socket\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414920 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-podinfo\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414952 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-data\") pod \"d729f15e-1443-40be-b0fc-4d9c3ae46844\" (UID: \"d729f15e-1443-40be-b0fc-4d9c3ae46844\") " Jul 10 13:38:02.415080 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.414960 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:38:02.415426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.415173 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:38:02.415426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.415292 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:38:02.415426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.415373 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.415426 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.415391 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.415613 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.415463 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d729f15e-1443-40be-b0fc-4d9c3ae46844-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:38:02.416014 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.415990 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-data" (OuterVolumeSpecName: "istio-data") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:38:02.417384 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.417355 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:38:02.417686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.417383 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-kube-api-access-vxmjx" (OuterVolumeSpecName: "kube-api-access-vxmjx") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "kube-api-access-vxmjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:38:02.417686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.417425 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-token" (OuterVolumeSpecName: "istio-token") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:38:02.417686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.417422 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d729f15e-1443-40be-b0fc-4d9c3ae46844" (UID: "d729f15e-1443-40be-b0fc-4d9c3ae46844"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:38:02.516176 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516149 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d729f15e-1443-40be-b0fc-4d9c3ae46844-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.516176 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516171 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.516176 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516181 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.516536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516189 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.516536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516197 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.516536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516204 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmjx\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-kube-api-access-vxmjx\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.516536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.516212 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d729f15e-1443-40be-b0fc-4d9c3ae46844-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:38:02.913481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.913452 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" Jul 10 13:38:02.913646 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.913492 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr" event={"ID":"d729f15e-1443-40be-b0fc-4d9c3ae46844","Type":"ContainerDied","Data":"2a55f81cd55459e13891367a6c6e225dc32aa025ed934c1837a8b848ef1a9d7f"} Jul 10 13:38:02.913646 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.913563 2567 scope.go:117] "RemoveContainer" containerID="2fccaeee418818c47bbf99093e439c3a1d981fa67de9f6aca2aaeae78e1db670" Jul 10 13:38:02.937791 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.937767 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr"] Jul 10 13:38:02.942580 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:02.942557 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-69fdb29c-openshift-default-86c45858b7-c25wr"] Jul 10 13:38:03.950359 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:03.950324 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" path="/var/lib/kubelet/pods/d729f15e-1443-40be-b0fc-4d9c3ae46844/volumes" Jul 10 13:38:07.553472 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:07.553441 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:07.625723 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:07.625684 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:14.523632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.523588 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:14.899256 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.899178 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8"] Jul 10 13:38:14.899551 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.899537 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" Jul 10 13:38:14.899599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.899553 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" Jul 10 13:38:14.899634 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.899625 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbbcc22-076a-400e-8514-5fe893d6a171" containerName="istio-proxy" Jul 10 13:38:14.899667 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.899636 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" Jul 10 13:38:14.899667 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.899642 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" Jul 10 13:38:14.903764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.903743 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:14.907587 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.907556 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-15039e79-openshift-default-dockercfg-9xfw6\"" Jul 10 13:38:14.925695 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.925674 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8"] Jul 10 13:38:14.930568 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:14.930541 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:15.005340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005302 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-credential-socket\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005338 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-data\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005542 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005401 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e300429a-13c5-410a-a33c-bec38e968b5a-istio-podinfo\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005542 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005459 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e300429a-13c5-410a-a33c-bec38e968b5a-istiod-ca-cert\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005542 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005499 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-istio-token\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005638 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005567 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-certs\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005638 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005585 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-socket\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005638 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005624 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-envoy\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.005732 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.005641 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47vph\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-kube-api-access-47vph\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.106809 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.106774 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-istio-token\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.106989 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.106866 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-certs\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.106989 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.106897 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-socket\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107065 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-envoy\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107065 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107059 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47vph\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-kube-api-access-47vph\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107141 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107110 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-credential-socket\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107146 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-data\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107229 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107215 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e300429a-13c5-410a-a33c-bec38e968b5a-istio-podinfo\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107276 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107246 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e300429a-13c5-410a-a33c-bec38e968b5a-istiod-ca-cert\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107328 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107274 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-certs\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107328 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107320 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-socket\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107513 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107485 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-credential-socket\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107619 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107601 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-data\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.107819 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.107803 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e300429a-13c5-410a-a33c-bec38e968b5a-istiod-ca-cert\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.109385 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.109353 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-envoy\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.109511 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.109495 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e300429a-13c5-410a-a33c-bec38e968b5a-istio-podinfo\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.115324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.115300 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-istio-token\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.115601 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.115582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47vph\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-kube-api-access-47vph\") pod \"e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.214115 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.214094 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:15.353447 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.353423 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8"] Jul 10 13:38:15.355279 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:38:15.355254 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode300429a_13c5_410a_a33c_bec38e968b5a.slice/crio-d45017f9746433707e3c6ace26170c83221242c0254ba7eea5d8390afcf2223f WatchSource:0}: Error finding container d45017f9746433707e3c6ace26170c83221242c0254ba7eea5d8390afcf2223f: Status 404 returned error can't find the container with id d45017f9746433707e3c6ace26170c83221242c0254ba7eea5d8390afcf2223f Jul 10 13:38:15.360579 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.360545 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:38:15.360686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.360612 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:38:15.360686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.360642 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:38:15.900063 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.900020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:15.982716 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.982677 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" event={"ID":"e300429a-13c5-410a-a33c-bec38e968b5a","Type":"ContainerStarted","Data":"2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440"} Jul 10 13:38:15.982941 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:15.982774 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" event={"ID":"e300429a-13c5-410a-a33c-bec38e968b5a","Type":"ContainerStarted","Data":"d45017f9746433707e3c6ace26170c83221242c0254ba7eea5d8390afcf2223f"} Jul 10 13:38:16.006306 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:16.006256 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podStartSLOduration=2.006243391 podStartE2EDuration="2.006243391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:38:16.004557638 +0000 UTC m=+1676.720935267" watchObservedRunningTime="2026-07-10 13:38:16.006243391 +0000 UTC m=+1676.722621072" Jul 10 13:38:16.214959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:16.214930 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:16.220105 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:16.220081 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:16.988721 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:16.988684 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:16.989562 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:16.989545 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:17.941597 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:17.941558 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:18.040257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:18.040225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:18.106921 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:18.106886 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:18.469330 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:18.469290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:18.623685 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:18.623651 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:18.723616 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:18.723525 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:22.490419 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:22.490380 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:31.990579 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:31.990541 2567 patch_prober.go:28] interesting pod/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 10 13:38:31.991095 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:31.990599 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 10 13:38:32.074358 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:32.074325 2567 generic.go:356] "Generic (PLEG): container finished" podID="e300429a-13c5-410a-a33c-bec38e968b5a" containerID="2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440" exitCode=0 Jul 10 13:38:32.074482 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:32.074401 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" event={"ID":"e300429a-13c5-410a-a33c-bec38e968b5a","Type":"ContainerDied","Data":"2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440"} Jul 10 13:38:32.074804 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:32.074788 2567 scope.go:117] "RemoveContainer" containerID="2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440" Jul 10 13:38:32.075720 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:32.075694 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:38:32.075785 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:32.075759 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:38:32.075824 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:32.075792 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:38:33.081265 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:33.081230 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" event={"ID":"e300429a-13c5-410a-a33c-bec38e968b5a","Type":"ContainerStarted","Data":"ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57"} Jul 10 13:38:33.215267 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:33.215222 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:33.215575 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:33.215537 2567 patch_prober.go:28] interesting pod/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 10 13:38:33.215662 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:33.215636 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 10 13:38:34.215011 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:34.214979 2567 patch_prober.go:28] interesting pod/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 10 13:38:34.215400 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:34.215028 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 10 13:38:35.214581 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:35.214546 2567 patch_prober.go:28] interesting pod/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 10 13:38:35.214753 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:35.214599 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 10 13:38:35.260883 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:35.260838 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:35.361724 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:35.361689 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:36.214628 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:36.214594 2567 patch_prober.go:28] interesting pod/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 10 13:38:36.214802 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:36.214645 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 10 13:38:37.214737 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:37.214706 2567 patch_prober.go:28] interesting pod/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 10 13:38:37.215144 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:37.214788 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 10 13:38:38.218911 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:38.218838 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:38.219314 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:38.219296 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:38.219935 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:38.219916 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:38:39.412020 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.411976 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:39.513112 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.513080 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:39.544362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.544336 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv"] Jul 10 13:38:39.544685 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.544673 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" Jul 10 13:38:39.544746 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.544686 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d729f15e-1443-40be-b0fc-4d9c3ae46844" containerName="istio-proxy" Jul 10 13:38:39.547921 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.547902 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.551626 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.551606 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-15039e79\"/\"kube-root-ca.crt\"" Jul 10 13:38:39.552814 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.552796 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-15039e79\"/\"e2e-lifecycle-model-15039e79-kserve-self-signed-certs\"" Jul 10 13:38:39.552937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.552857 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-15039e79\"/\"default-dockercfg-gvvdl\"" Jul 10 13:38:39.552937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.552913 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-15039e79\"/\"openshift-service-ca.crt\"" Jul 10 13:38:39.562835 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.562811 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv"] Jul 10 13:38:39.611981 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.611961 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:39.625170 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.625145 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j72wr\" (UniqueName: \"kubernetes.io/projected/bb749467-865e-46b8-b80e-538681bc7ce5-kube-api-access-j72wr\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.625269 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.625178 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-tmp-dir\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.625269 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.625201 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb749467-865e-46b8-b80e-538681bc7ce5-tls-certs\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.625345 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.625277 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-home\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.625345 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.625300 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-dshm\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.625345 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.625322 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-model-cache\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726174 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726148 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j72wr\" (UniqueName: \"kubernetes.io/projected/bb749467-865e-46b8-b80e-538681bc7ce5-kube-api-access-j72wr\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726182 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-tmp-dir\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726204 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb749467-865e-46b8-b80e-538681bc7ce5-tls-certs\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726233 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-home\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726249 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-dshm\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726266 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-model-cache\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726559 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726530 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-tmp-dir\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726618 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726582 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-home\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.726687 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.726664 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-model-cache\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.728528 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.728507 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-dshm\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.728673 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.728656 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb749467-865e-46b8-b80e-538681bc7ce5-tls-certs\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.733673 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.733655 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j72wr\" (UniqueName: \"kubernetes.io/projected/bb749467-865e-46b8-b80e-538681bc7ce5-kube-api-access-j72wr\") pod \"e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.858128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.858095 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:39.994252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:39.994226 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv"] Jul 10 13:38:39.995247 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:38:39.995223 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb749467_865e_46b8_b80e_538681bc7ce5.slice/crio-3b38be51fbe0b146f433e1ccf02dddddced948160d0a6a905ea79ee25ca23701 WatchSource:0}: Error finding container 3b38be51fbe0b146f433e1ccf02dddddced948160d0a6a905ea79ee25ca23701: Status 404 returned error can't find the container with id 3b38be51fbe0b146f433e1ccf02dddddced948160d0a6a905ea79ee25ca23701 Jul 10 13:38:40.117299 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:40.117265 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" event={"ID":"bb749467-865e-46b8-b80e-538681bc7ce5","Type":"ContainerStarted","Data":"6e3f4242e88354fd22cbcebfec012b2bbe6db63e0918bf47056baceec2338941"} Jul 10 13:38:40.117421 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:40.117314 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" event={"ID":"bb749467-865e-46b8-b80e-538681bc7ce5","Type":"ContainerStarted","Data":"3b38be51fbe0b146f433e1ccf02dddddced948160d0a6a905ea79ee25ca23701"} Jul 10 13:38:40.136155 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:40.136108 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" podStartSLOduration=1.136094239 podStartE2EDuration="1.136094239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:38:40.134118766 +0000 UTC m=+1700.850496404" watchObservedRunningTime="2026-07-10 13:38:40.136094239 +0000 UTC m=+1700.852471868" Jul 10 13:38:47.425675 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:47.425589 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:38:49.859146 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:49.859112 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:49.859609 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:49.859157 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:49.871489 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:49.871462 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:38:50.177520 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:38:50.177442 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:39:00.053324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:00.053290 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:00.479578 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:00.479547 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:06.129421 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:06.129378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:11.113010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:11.112971 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv"] Jul 10 13:39:11.204360 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:11.204326 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:26.136046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.135990 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" podUID="bb749467-865e-46b8-b80e-538681bc7ce5" containerName="main" containerID="cri-o://6e3f4242e88354fd22cbcebfec012b2bbe6db63e0918bf47056baceec2338941" gracePeriod=45 Jul 10 13:39:26.345445 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.345416 2567 generic.go:356] "Generic (PLEG): container finished" podID="bb749467-865e-46b8-b80e-538681bc7ce5" containerID="6e3f4242e88354fd22cbcebfec012b2bbe6db63e0918bf47056baceec2338941" exitCode=0 Jul 10 13:39:26.345576 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.345491 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" event={"ID":"bb749467-865e-46b8-b80e-538681bc7ce5","Type":"ContainerDied","Data":"6e3f4242e88354fd22cbcebfec012b2bbe6db63e0918bf47056baceec2338941"} Jul 10 13:39:26.384793 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.384771 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:39:26.508706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.508674 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-home\") pod \"bb749467-865e-46b8-b80e-538681bc7ce5\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " Jul 10 13:39:26.508706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.508711 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-tmp-dir\") pod \"bb749467-865e-46b8-b80e-538681bc7ce5\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " Jul 10 13:39:26.508993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.508731 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-dshm\") pod \"bb749467-865e-46b8-b80e-538681bc7ce5\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " Jul 10 13:39:26.508993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.508769 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb749467-865e-46b8-b80e-538681bc7ce5-tls-certs\") pod \"bb749467-865e-46b8-b80e-538681bc7ce5\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " Jul 10 13:39:26.508993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.508908 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j72wr\" (UniqueName: \"kubernetes.io/projected/bb749467-865e-46b8-b80e-538681bc7ce5-kube-api-access-j72wr\") pod \"bb749467-865e-46b8-b80e-538681bc7ce5\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " Jul 10 13:39:26.508993 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.508943 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-model-cache\") pod \"bb749467-865e-46b8-b80e-538681bc7ce5\" (UID: \"bb749467-865e-46b8-b80e-538681bc7ce5\") " Jul 10 13:39:26.509193 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.509095 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-home" (OuterVolumeSpecName: "home") pod "bb749467-865e-46b8-b80e-538681bc7ce5" (UID: "bb749467-865e-46b8-b80e-538681bc7ce5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:26.509343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.509311 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:26.509575 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.509524 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb749467-865e-46b8-b80e-538681bc7ce5" (UID: "bb749467-865e-46b8-b80e-538681bc7ce5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:26.509762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.509725 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb749467-865e-46b8-b80e-538681bc7ce5" (UID: "bb749467-865e-46b8-b80e-538681bc7ce5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:26.511597 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.511571 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-dshm" (OuterVolumeSpecName: "dshm") pod "bb749467-865e-46b8-b80e-538681bc7ce5" (UID: "bb749467-865e-46b8-b80e-538681bc7ce5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:26.511699 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.511617 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb749467-865e-46b8-b80e-538681bc7ce5-kube-api-access-j72wr" (OuterVolumeSpecName: "kube-api-access-j72wr") pod "bb749467-865e-46b8-b80e-538681bc7ce5" (UID: "bb749467-865e-46b8-b80e-538681bc7ce5"). InnerVolumeSpecName "kube-api-access-j72wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:39:26.511699 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.511622 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb749467-865e-46b8-b80e-538681bc7ce5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb749467-865e-46b8-b80e-538681bc7ce5" (UID: "bb749467-865e-46b8-b80e-538681bc7ce5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:39:26.610446 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.610417 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j72wr\" (UniqueName: \"kubernetes.io/projected/bb749467-865e-46b8-b80e-538681bc7ce5-kube-api-access-j72wr\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:26.610446 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.610441 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:26.610587 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.610452 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:26.610587 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.610460 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb749467-865e-46b8-b80e-538681bc7ce5-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:26.610587 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:26.610467 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb749467-865e-46b8-b80e-538681bc7ce5-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:27.351817 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:27.351789 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" Jul 10 13:39:27.352191 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:27.351788 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv" event={"ID":"bb749467-865e-46b8-b80e-538681bc7ce5","Type":"ContainerDied","Data":"3b38be51fbe0b146f433e1ccf02dddddced948160d0a6a905ea79ee25ca23701"} Jul 10 13:39:27.352944 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:27.351912 2567 scope.go:117] "RemoveContainer" containerID="6e3f4242e88354fd22cbcebfec012b2bbe6db63e0918bf47056baceec2338941" Jul 10 13:39:27.374224 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:27.374202 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv"] Jul 10 13:39:27.378799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:27.378778 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-15039e79/e2e-lifecycle-model-15039e79-kserve-8544956bcf-5j6jv"] Jul 10 13:39:27.951245 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:27.951214 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb749467-865e-46b8-b80e-538681bc7ce5" path="/var/lib/kubelet/pods/bb749467-865e-46b8-b80e-538681bc7ce5/volumes" Jul 10 13:39:47.965160 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:47.965119 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:48.340549 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:48.340459 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:53.949101 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:53.949042 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:54.013048 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:54.013019 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:54.204679 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:54.204607 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8"] Jul 10 13:39:54.204836 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:54.204816 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" containerID="cri-o://ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57" gracePeriod=30 Jul 10 13:39:54.229563 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:54.229526 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:54.468357 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:54.468276 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:55.547706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:55.547664 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:55.641884 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:55.641858 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:55.740596 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:55.740546 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:56.443503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:56.443470 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:56.539587 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:56.539551 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:56.639446 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:56.639416 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:57.969433 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:57.969405 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:39:59.448333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.448312 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:39:59.516303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.516238 2567 generic.go:356] "Generic (PLEG): container finished" podID="e300429a-13c5-410a-a33c-bec38e968b5a" containerID="ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57" exitCode=0 Jul 10 13:39:59.516303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.516284 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" Jul 10 13:39:59.516303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.516285 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" event={"ID":"e300429a-13c5-410a-a33c-bec38e968b5a","Type":"ContainerDied","Data":"ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57"} Jul 10 13:39:59.516513 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.516331 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8" event={"ID":"e300429a-13c5-410a-a33c-bec38e968b5a","Type":"ContainerDied","Data":"d45017f9746433707e3c6ace26170c83221242c0254ba7eea5d8390afcf2223f"} Jul 10 13:39:59.516513 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.516357 2567 scope.go:117] "RemoveContainer" containerID="ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57" Jul 10 13:39:59.524632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.524614 2567 scope.go:117] "RemoveContainer" containerID="2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440" Jul 10 13:39:59.532255 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.532238 2567 scope.go:117] "RemoveContainer" containerID="ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57" Jul 10 13:39:59.532492 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:39:59.532475 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57\": container with ID starting with ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57 not found: ID does not exist" containerID="ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57" Jul 10 13:39:59.532542 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.532502 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57"} err="failed to get container status \"ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57\": rpc error: code = NotFound desc = could not find container \"ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57\": container with ID starting with ece028aa297acb98a4ebd8c956c750d862d84c75bb0d14c6896add3901e96d57 not found: ID does not exist" Jul 10 13:39:59.532542 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.532519 2567 scope.go:117] "RemoveContainer" containerID="2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440" Jul 10 13:39:59.532752 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:39:59.532734 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440\": container with ID starting with 2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440 not found: ID does not exist" containerID="2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440" Jul 10 13:39:59.532796 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.532759 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440"} err="failed to get container status \"2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440\": rpc error: code = NotFound desc = could not find container \"2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440\": container with ID starting with 2352ae3edd6c1ffd95284616ee45b3ba9f8e972814cdfb30b8cd0fa8d3fa6440 not found: ID does not exist" Jul 10 13:39:59.564186 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564164 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-data\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564266 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564199 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e300429a-13c5-410a-a33c-bec38e968b5a-istio-podinfo\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564266 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564241 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47vph\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-kube-api-access-47vph\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564266 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564259 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-socket\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564365 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564286 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e300429a-13c5-410a-a33c-bec38e968b5a-istiod-ca-cert\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564365 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564303 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-credential-socket\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564365 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564346 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-istio-token\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564509 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564420 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-certs\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564509 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564472 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-envoy\") pod \"e300429a-13c5-410a-a33c-bec38e968b5a\" (UID: \"e300429a-13c5-410a-a33c-bec38e968b5a\") " Jul 10 13:39:59.564634 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564604 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:59.564693 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564657 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e300429a-13c5-410a-a33c-bec38e968b5a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:39:59.564830 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564807 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e300429a-13c5-410a-a33c-bec38e968b5a-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.564919 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564840 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.564919 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564889 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:59.565006 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.564936 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:59.565084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.565067 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:59.566663 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.566636 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e300429a-13c5-410a-a33c-bec38e968b5a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:39:59.566791 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.566770 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:39:59.566894 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.566789 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-kube-api-access-47vph" (OuterVolumeSpecName: "kube-api-access-47vph") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "kube-api-access-47vph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:39:59.566894 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.566824 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e300429a-13c5-410a-a33c-bec38e968b5a" (UID: "e300429a-13c5-410a-a33c-bec38e968b5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:39:59.666268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666240 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.666268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666265 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.666407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666279 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.666407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666290 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.666407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666300 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e300429a-13c5-410a-a33c-bec38e968b5a-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.666407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666312 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47vph\" (UniqueName: \"kubernetes.io/projected/e300429a-13c5-410a-a33c-bec38e968b5a-kube-api-access-47vph\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.666407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.666326 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e300429a-13c5-410a-a33c-bec38e968b5a-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:39:59.838939 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.838912 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8"] Jul 10 13:39:59.843407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.843387 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15039e79-openshift-default-fbc5c77c5-nfvx8"] Jul 10 13:39:59.952148 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:39:59.952119 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" path="/var/lib/kubelet/pods/e300429a-13c5-410a-a33c-bec38e968b5a/volumes" Jul 10 13:40:03.643379 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:03.643339 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:03.746810 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:03.746780 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:04.314241 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:04.314208 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:10.007145 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:10.007105 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:11.219174 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:11.219039 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:19.909631 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:19.909595 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:40:19.919815 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:19.919794 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:40:20.688114 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:20.688076 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:21.049506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:21.049477 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:42.046216 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:42.046181 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:48.203381 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.203346 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:48.312627 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.312595 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:48.590216 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.590132 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:48.895749 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.895677 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2"] Jul 10 13:40:48.896062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896048 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" Jul 10 13:40:48.896109 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896064 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" Jul 10 13:40:48.896109 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896073 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb749467-865e-46b8-b80e-538681bc7ce5" containerName="main" Jul 10 13:40:48.896109 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896079 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb749467-865e-46b8-b80e-538681bc7ce5" containerName="main" Jul 10 13:40:48.896109 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896090 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" Jul 10 13:40:48.896109 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896095 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" Jul 10 13:40:48.896280 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896151 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" Jul 10 13:40:48.896280 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896158 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e300429a-13c5-410a-a33c-bec38e968b5a" containerName="istio-proxy" Jul 10 13:40:48.896280 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.896167 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb749467-865e-46b8-b80e-538681bc7ce5" containerName="main" Jul 10 13:40:48.899261 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.899235 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.902302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.902280 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c7549e-openshift-default-dockercfg-cnfgv\"" Jul 10 13:40:48.916337 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.916312 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2"] Jul 10 13:40:48.923194 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.923144 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:48.990413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990365 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6454d6a-f4df-4052-a1fb-021223bbe30b-istiod-ca-cert\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990442 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-data\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990533 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-credential-socket\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990590 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-certs\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990708 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990645 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-token\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990817 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990724 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-socket\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990817 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990750 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-envoy\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990817 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990778 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-podinfo\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:48.990996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:48.990813 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpws\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-kube-api-access-zgpws\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.030084 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.030040 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:49.091959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.091915 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-podinfo\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.091965 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpws\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-kube-api-access-zgpws\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092002 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6454d6a-f4df-4052-a1fb-021223bbe30b-istiod-ca-cert\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092022 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-data\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092066 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-credential-socket\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092307 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092178 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-certs\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092307 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092227 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-token\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092309 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-socket\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092334 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-envoy\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092523 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092471 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-credential-socket\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092578 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092524 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-data\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092585 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-certs\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092811 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092792 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-socket\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.092931 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.092868 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6454d6a-f4df-4052-a1fb-021223bbe30b-istiod-ca-cert\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.094469 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.094442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-podinfo\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.094580 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.094525 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-envoy\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.100302 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.100282 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-token\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.101713 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.101684 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpws\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-kube-api-access-zgpws\") pod \"e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.209370 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.209347 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:49.347534 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.347504 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2"] Jul 10 13:40:49.348324 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:40:49.348290 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6454d6a_f4df_4052_a1fb_021223bbe30b.slice/crio-b3467b992e77b79238f64685c6742a070302063154711a1c09c46dfc496065e5 WatchSource:0}: Error finding container b3467b992e77b79238f64685c6742a070302063154711a1c09c46dfc496065e5: Status 404 returned error can't find the container with id b3467b992e77b79238f64685c6742a070302063154711a1c09c46dfc496065e5 Jul 10 13:40:49.350464 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.350433 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:40:49.350547 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.350497 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:40:49.350547 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.350528 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:40:49.771344 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.771304 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" event={"ID":"d6454d6a-f4df-4052-a1fb-021223bbe30b","Type":"ContainerStarted","Data":"d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887"} Jul 10 13:40:49.771532 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.771353 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" event={"ID":"d6454d6a-f4df-4052-a1fb-021223bbe30b","Type":"ContainerStarted","Data":"b3467b992e77b79238f64685c6742a070302063154711a1c09c46dfc496065e5"} Jul 10 13:40:49.794802 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:49.794746 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podStartSLOduration=1.794730558 podStartE2EDuration="1.794730558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:40:49.792454294 +0000 UTC m=+1830.508831922" watchObservedRunningTime="2026-07-10 13:40:49.794730558 +0000 UTC m=+1830.511108183" Jul 10 13:40:50.210128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:50.210101 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:50.214612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:50.214590 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:50.777350 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:50.777318 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:50.778387 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:50.778370 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:40:51.993858 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:51.993804 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:52.151280 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:52.151247 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:52.253395 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:52.253296 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:52.745256 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:52.745224 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:52.844620 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:52.844591 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:56.304912 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.304878 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:56.653723 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.653635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:56.956037 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.956005 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg"] Jul 10 13:40:56.959637 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.959616 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:56.962136 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.962115 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-faacd6-openshift-default-dockercfg-67f2w\"" Jul 10 13:40:56.971935 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.971915 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg"] Jul 10 13:40:56.984554 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:56.984530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:57.060820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.060785 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-podinfo\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061019 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.060834 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-data\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061019 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.060944 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-certs\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061019 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.061003 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-token\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.061054 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-socket\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.061095 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-credential-socket\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.061121 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-envoy\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.061144 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51607bc-1d6a-4d88-a350-75f2caf259b0-istiod-ca-cert\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.061204 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.061174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brs6j\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-kube-api-access-brs6j\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162008 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-token\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162073 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-socket\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162110 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-credential-socket\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162134 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-envoy\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162158 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51607bc-1d6a-4d88-a350-75f2caf259b0-istiod-ca-cert\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162184 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brs6j\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-kube-api-access-brs6j\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162242 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162241 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-podinfo\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162607 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162542 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-credential-socket\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162602 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-socket\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162724 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-data\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162724 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-data\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162829 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162763 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-certs\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.162904 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.162863 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51607bc-1d6a-4d88-a350-75f2caf259b0-istiod-ca-cert\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.163053 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.163033 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-certs\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.164429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.164411 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-envoy\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.164668 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.164650 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-podinfo\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.169464 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.169442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brs6j\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-kube-api-access-brs6j\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.169566 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.169500 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-token\") pod \"e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.184427 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.184398 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:57.270801 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.270728 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:57.414037 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.414014 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg"] Jul 10 13:40:57.414906 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:40:57.414881 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda51607bc_1d6a_4d88_a350_75f2caf259b0.slice/crio-af1179dd7bb96d7eaa74a028db9bcd6cf05ed5d147b13169d0139940b48937f8 WatchSource:0}: Error finding container af1179dd7bb96d7eaa74a028db9bcd6cf05ed5d147b13169d0139940b48937f8: Status 404 returned error can't find the container with id af1179dd7bb96d7eaa74a028db9bcd6cf05ed5d147b13169d0139940b48937f8 Jul 10 13:40:57.417103 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.417071 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:40:57.417182 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.417130 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:40:57.417182 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.417156 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:40:57.816609 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.816577 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" event={"ID":"a51607bc-1d6a-4d88-a350-75f2caf259b0","Type":"ContainerStarted","Data":"cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080"} Jul 10 13:40:57.816792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.816628 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" event={"ID":"a51607bc-1d6a-4d88-a350-75f2caf259b0","Type":"ContainerStarted","Data":"af1179dd7bb96d7eaa74a028db9bcd6cf05ed5d147b13169d0139940b48937f8"} Jul 10 13:40:57.839434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:57.839387 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podStartSLOduration=1.839371455 podStartE2EDuration="1.839371455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:40:57.836308426 +0000 UTC m=+1838.552686073" watchObservedRunningTime="2026-07-10 13:40:57.839371455 +0000 UTC m=+1838.555749081" Jul 10 13:40:58.271757 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:58.271730 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:58.276492 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:58.276468 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:58.821968 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:58.821941 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:58.822870 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:58.822827 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:40:59.797492 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:59.797460 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:40:59.887991 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:40:59.887955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:00.082133 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:00.082045 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:00.179547 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:00.179518 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:00.719658 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:00.719626 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:00.819669 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:00.819636 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:04.334639 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:04.334596 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:06.820006 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:06.819968 2567 patch_prober.go:28] interesting pod/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 13:41:06.820416 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:06.820032 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 13:41:07.879227 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:07.879197 2567 generic.go:356] "Generic (PLEG): container finished" podID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerID="d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887" exitCode=0 Jul 10 13:41:07.879608 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:07.879233 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" event={"ID":"d6454d6a-f4df-4052-a1fb-021223bbe30b","Type":"ContainerDied","Data":"d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887"} Jul 10 13:41:07.879608 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:07.879601 2567 scope.go:117] "RemoveContainer" containerID="d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887" Jul 10 13:41:07.880569 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:07.880539 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:41:07.880683 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:07.880609 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:41:07.880683 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:07.880645 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:41:08.887751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:08.887712 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" event={"ID":"d6454d6a-f4df-4052-a1fb-021223bbe30b","Type":"ContainerStarted","Data":"e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c"} Jul 10 13:41:09.210154 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:09.210127 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:41:09.210446 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:09.210424 2567 patch_prober.go:28] interesting pod/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 10 13:41:09.210491 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:09.210469 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 10 13:41:10.210096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:10.210064 2567 patch_prober.go:28] interesting pod/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 10 13:41:10.210473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:10.210110 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 10 13:41:11.045140 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.045102 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks"] Jul 10 13:41:11.056230 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.056204 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.063230 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.063204 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7549e\"/\"default-dockercfg-qnxm2\"" Jul 10 13:41:11.063677 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.063658 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7549e\"/\"kube-root-ca.crt\"" Jul 10 13:41:11.064021 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.064005 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c7549e\"/\"test-model-c7549e-kserve-self-signed-certs\"" Jul 10 13:41:11.064370 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.064351 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c7549e\"/\"openshift-service-ca.crt\"" Jul 10 13:41:11.082943 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.082912 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:11.090471 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.090407 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks"] Jul 10 13:41:11.094456 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.094346 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tls-certs\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.094456 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.094427 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tmp-dir\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.094643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.094498 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-model-cache\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.094643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.094536 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4vr\" (UniqueName: \"kubernetes.io/projected/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-kube-api-access-kj4vr\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.094643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.094565 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-home\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.094643 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.094606 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-dshm\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.195742 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.195698 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tls-certs\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.195742 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.195744 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tmp-dir\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.195807 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-model-cache\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.195868 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4vr\" (UniqueName: \"kubernetes.io/projected/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-kube-api-access-kj4vr\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.195895 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-home\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196044 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.195933 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-dshm\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196309 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.196287 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-model-cache\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196375 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.196352 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tmp-dir\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.196427 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.196408 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-home\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.198127 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.198105 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-dshm\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.198402 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.198386 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tls-certs\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.205136 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.205114 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4vr\" (UniqueName: \"kubernetes.io/projected/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-kube-api-access-kj4vr\") pod \"test-model-c7549e-kserve-5cd594bf47-s89ks\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.209600 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.209573 2567 patch_prober.go:28] interesting pod/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 10 13:41:11.209683 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.209617 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 10 13:41:11.391134 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.391034 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:11.625947 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.625920 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks"] Jul 10 13:41:11.627583 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:41:11.627554 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24ba50b9_6da6_44ef_bb8e_1bb604bc59dc.slice/crio-35670184c16f2e92f5a8fa183b34436809506561ced262f53c92201ef6659fde WatchSource:0}: Error finding container 35670184c16f2e92f5a8fa183b34436809506561ced262f53c92201ef6659fde: Status 404 returned error can't find the container with id 35670184c16f2e92f5a8fa183b34436809506561ced262f53c92201ef6659fde Jul 10 13:41:11.912432 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.912342 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" event={"ID":"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc","Type":"ContainerStarted","Data":"6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050"} Jul 10 13:41:11.912432 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:11.912389 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" event={"ID":"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc","Type":"ContainerStarted","Data":"35670184c16f2e92f5a8fa183b34436809506561ced262f53c92201ef6659fde"} Jul 10 13:41:12.209917 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:12.209881 2567 patch_prober.go:28] interesting pod/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 10 13:41:12.210065 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:12.209945 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 10 13:41:13.214390 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:13.214360 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:41:13.214936 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:13.214863 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:41:13.215632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:13.215615 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:41:13.240250 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:13.240207 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" podStartSLOduration=2.240195112 podStartE2EDuration="2.240195112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:41:11.943974614 +0000 UTC m=+1852.660352242" watchObservedRunningTime="2026-07-10 13:41:13.240195112 +0000 UTC m=+1853.956572737" Jul 10 13:41:13.641318 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:13.641231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:13.744268 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:13.744231 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:14.406892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.406346 2567 patch_prober.go:28] interesting pod/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:54920->10.133.0.41:15021: read: connection reset by peer" start-of-body= Jul 10 13:41:14.406892 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.406424 2567 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:54920->10.133.0.41:15021: read: connection reset by peer" Jul 10 13:41:14.937864 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.937817 2567 generic.go:356] "Generic (PLEG): container finished" podID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerID="cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080" exitCode=0 Jul 10 13:41:14.938063 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.937896 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" event={"ID":"a51607bc-1d6a-4d88-a350-75f2caf259b0","Type":"ContainerDied","Data":"cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080"} Jul 10 13:41:14.938412 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.938396 2567 scope.go:117] "RemoveContainer" containerID="cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080" Jul 10 13:41:14.939305 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.939282 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:41:14.939360 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.939341 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:41:14.939401 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:14.939372 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:41:15.961444 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:15.961401 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" event={"ID":"a51607bc-1d6a-4d88-a350-75f2caf259b0","Type":"ContainerStarted","Data":"f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9"} Jul 10 13:41:16.271692 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:16.271599 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:41:16.271857 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:16.271775 2567 patch_prober.go:28] interesting pod/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 10 13:41:16.271914 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:16.271828 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 10 13:41:17.271294 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:17.271249 2567 patch_prober.go:28] interesting pod/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 10 13:41:17.271766 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:17.271314 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 10 13:41:18.271456 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:18.271421 2567 patch_prober.go:28] interesting pod/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 10 13:41:18.271888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:18.271502 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 10 13:41:19.271982 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:19.271944 2567 patch_prober.go:28] interesting pod/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 10 13:41:19.272385 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:19.272001 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 10 13:41:20.274501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:20.274474 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:41:20.274913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:20.274749 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:41:20.275451 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:20.275432 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:41:21.391827 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:21.391782 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:21.392391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:21.391872 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:21.404494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:21.404463 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:22.003632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:22.003598 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:41:22.275697 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:22.275609 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:22.378096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:22.378059 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:43.453011 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:43.452900 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:41:44.925390 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:41:44.925342 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:44.678536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:44.678498 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks"] Jul 10 13:42:44.698413 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:44.698375 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:45.650937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:45.650906 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:45.855220 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:45.855183 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:46.481834 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:46.481792 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:46.694303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:46.694269 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:46.801088 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:46.801011 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:47.062051 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:47.061962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:51.521897 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:51.521864 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:42:59.698778 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:59.698741 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" podUID="24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" containerName="main" containerID="cri-o://6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050" gracePeriod=45 Jul 10 13:42:59.952095 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:42:59.952039 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:43:00.056169 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056142 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-dshm\") pod \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " Jul 10 13:43:00.056314 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056243 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-home\") pod \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " Jul 10 13:43:00.056314 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056282 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj4vr\" (UniqueName: \"kubernetes.io/projected/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-kube-api-access-kj4vr\") pod \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " Jul 10 13:43:00.056400 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056321 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-model-cache\") pod \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " Jul 10 13:43:00.056400 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056369 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tmp-dir\") pod \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " Jul 10 13:43:00.056499 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056398 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tls-certs\") pod \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\" (UID: \"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc\") " Jul 10 13:43:00.056603 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056574 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-home" (OuterVolumeSpecName: "home") pod "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" (UID: "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:00.056679 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056581 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" (UID: "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:00.056679 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056655 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" (UID: "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:00.056785 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056771 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:00.056835 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056791 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:00.056835 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.056806 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:00.058495 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.058474 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-kube-api-access-kj4vr" (OuterVolumeSpecName: "kube-api-access-kj4vr") pod "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" (UID: "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc"). InnerVolumeSpecName "kube-api-access-kj4vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:43:00.058495 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.058478 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-dshm" (OuterVolumeSpecName: "dshm") pod "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" (UID: "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:00.058759 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.058736 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" (UID: "24ba50b9-6da6-44ef-bb8e-1bb604bc59dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:43:00.157358 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.157334 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:00.157358 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.157355 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:00.157481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.157364 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4vr\" (UniqueName: \"kubernetes.io/projected/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc-kube-api-access-kj4vr\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:00.501352 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.501318 2567 generic.go:356] "Generic (PLEG): container finished" podID="24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" containerID="6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050" exitCode=0 Jul 10 13:43:00.501503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.501356 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" Jul 10 13:43:00.501503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.501401 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" event={"ID":"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc","Type":"ContainerDied","Data":"6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050"} Jul 10 13:43:00.501503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.501443 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks" event={"ID":"24ba50b9-6da6-44ef-bb8e-1bb604bc59dc","Type":"ContainerDied","Data":"35670184c16f2e92f5a8fa183b34436809506561ced262f53c92201ef6659fde"} Jul 10 13:43:00.501503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.501461 2567 scope.go:117] "RemoveContainer" containerID="6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050" Jul 10 13:43:00.511005 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.510988 2567 scope.go:117] "RemoveContainer" containerID="6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050" Jul 10 13:43:00.511246 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:43:00.511229 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050\": container with ID starting with 6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050 not found: ID does not exist" containerID="6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050" Jul 10 13:43:00.511295 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.511253 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050"} err="failed to get container status \"6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050\": rpc error: code = NotFound desc = could not find container \"6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050\": container with ID starting with 6cc37509e571d523531fc24cbac3aae9eda81a9ff5d0857ae9a0ac8f5e9fa050 not found: ID does not exist" Jul 10 13:43:00.533488 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.533464 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks"] Jul 10 13:43:00.535520 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:00.535500 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c7549e/test-model-c7549e-kserve-5cd594bf47-s89ks"] Jul 10 13:43:01.950926 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:01.950897 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" path="/var/lib/kubelet/pods/24ba50b9-6da6-44ef-bb8e-1bb604bc59dc/volumes" Jul 10 13:43:28.544151 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:28.544119 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:28.787272 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:28.787235 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2"] Jul 10 13:43:28.787519 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:28.787491 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" containerID="cri-o://e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c" gracePeriod=30 Jul 10 13:43:28.810295 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:28.810217 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:29.046976 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:29.046942 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:29.210706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:29.210673 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:29.630288 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:29.630194 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:29.730805 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:29.730766 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:29.908275 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:29.908189 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:34.036521 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.036499 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:43:34.135807 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135740 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6454d6a-f4df-4052-a1fb-021223bbe30b-istiod-ca-cert\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.135807 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135781 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-certs\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.135807 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135807 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-socket\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135831 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-data\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135891 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgpws\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-kube-api-access-zgpws\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135937 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-podinfo\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.135972 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-envoy\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136061 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-token\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136090 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136066 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:34.136399 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136116 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:34.136399 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136157 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-credential-socket\") pod \"d6454d6a-f4df-4052-a1fb-021223bbe30b\" (UID: \"d6454d6a-f4df-4052-a1fb-021223bbe30b\") " Jul 10 13:43:34.136399 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136163 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6454d6a-f4df-4052-a1fb-021223bbe30b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:43:34.136604 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136410 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:34.136764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136744 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6454d6a-f4df-4052-a1fb-021223bbe30b-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.136866 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136770 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.136866 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136797 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.136866 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136809 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.136866 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.136840 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:34.138141 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.138116 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:43:34.138251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.138229 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:43:34.138495 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.138479 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:34.138556 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.138503 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-kube-api-access-zgpws" (OuterVolumeSpecName: "kube-api-access-zgpws") pod "d6454d6a-f4df-4052-a1fb-021223bbe30b" (UID: "d6454d6a-f4df-4052-a1fb-021223bbe30b"). InnerVolumeSpecName "kube-api-access-zgpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:43:34.237755 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.237729 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.237755 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.237751 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.237913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.237761 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpws\" (UniqueName: \"kubernetes.io/projected/d6454d6a-f4df-4052-a1fb-021223bbe30b-kube-api-access-zgpws\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.237913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.237770 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.237913 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.237778 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6454d6a-f4df-4052-a1fb-021223bbe30b-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:34.677440 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.677404 2567 generic.go:356] "Generic (PLEG): container finished" podID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerID="e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c" exitCode=0 Jul 10 13:43:34.677617 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.677450 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" event={"ID":"d6454d6a-f4df-4052-a1fb-021223bbe30b","Type":"ContainerDied","Data":"e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c"} Jul 10 13:43:34.677617 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.677461 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" Jul 10 13:43:34.677617 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.677520 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2" event={"ID":"d6454d6a-f4df-4052-a1fb-021223bbe30b","Type":"ContainerDied","Data":"b3467b992e77b79238f64685c6742a070302063154711a1c09c46dfc496065e5"} Jul 10 13:43:34.677617 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.677544 2567 scope.go:117] "RemoveContainer" containerID="e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c" Jul 10 13:43:34.687182 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.687163 2567 scope.go:117] "RemoveContainer" containerID="d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887" Jul 10 13:43:34.694765 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.694746 2567 scope.go:117] "RemoveContainer" containerID="e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c" Jul 10 13:43:34.695006 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:43:34.694987 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c\": container with ID starting with e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c not found: ID does not exist" containerID="e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c" Jul 10 13:43:34.695092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.695012 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c"} err="failed to get container status \"e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c\": rpc error: code = NotFound desc = could not find container \"e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c\": container with ID starting with e4982043e1f456d996692c5c0d2462ff70fd40e86aa0fe2990d953859671d31c not found: ID does not exist" Jul 10 13:43:34.695092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.695028 2567 scope.go:117] "RemoveContainer" containerID="d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887" Jul 10 13:43:34.695262 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:43:34.695245 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887\": container with ID starting with d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887 not found: ID does not exist" containerID="d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887" Jul 10 13:43:34.695301 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.695269 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887"} err="failed to get container status \"d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887\": rpc error: code = NotFound desc = could not find container \"d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887\": container with ID starting with d9c191233dd3b2647fa63d1889174a0db6597d7465933cc82891fd7928159887 not found: ID does not exist" Jul 10 13:43:34.711133 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.711103 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2"] Jul 10 13:43:34.717056 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:34.717036 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c7549e-openshift-default-cc74dd759-pshn2"] Jul 10 13:43:35.951809 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:35.951764 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" path="/var/lib/kubelet/pods/d6454d6a-f4df-4052-a1fb-021223bbe30b/volumes" Jul 10 13:43:35.991262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:35.991225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:36.094409 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.094372 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:36.189942 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.189912 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:36.289436 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.289348 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:36.448780 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.448749 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg"] Jul 10 13:43:36.449015 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.448972 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" containerID="cri-o://f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9" gracePeriod=30 Jul 10 13:43:36.467371 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.467348 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:36.716633 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:36.716603 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:37.080982 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:37.080897 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:37.423584 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:37.423492 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:37.627452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:37.627416 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:37.957777 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:37.957747 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:41.696623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.696601 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:43:41.716326 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.716213 2567 generic.go:356] "Generic (PLEG): container finished" podID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerID="f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9" exitCode=0 Jul 10 13:43:41.716326 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.716281 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" Jul 10 13:43:41.717163 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.716338 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" event={"ID":"a51607bc-1d6a-4d88-a350-75f2caf259b0","Type":"ContainerDied","Data":"f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9"} Jul 10 13:43:41.717163 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.716393 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg" event={"ID":"a51607bc-1d6a-4d88-a350-75f2caf259b0","Type":"ContainerDied","Data":"af1179dd7bb96d7eaa74a028db9bcd6cf05ed5d147b13169d0139940b48937f8"} Jul 10 13:43:41.717163 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.716428 2567 scope.go:117] "RemoveContainer" containerID="f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9" Jul 10 13:43:41.726048 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.726025 2567 scope.go:117] "RemoveContainer" containerID="cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080" Jul 10 13:43:41.733656 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.733641 2567 scope.go:117] "RemoveContainer" containerID="f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9" Jul 10 13:43:41.733918 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:43:41.733892 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9\": container with ID starting with f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9 not found: ID does not exist" containerID="f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9" Jul 10 13:43:41.734029 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.733926 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9"} err="failed to get container status \"f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9\": rpc error: code = NotFound desc = could not find container \"f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9\": container with ID starting with f66bcea8124e14844a0264ff9d619ef23469323ad993f09d4c43da0490ddd8f9 not found: ID does not exist" Jul 10 13:43:41.734029 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.733945 2567 scope.go:117] "RemoveContainer" containerID="cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080" Jul 10 13:43:41.734165 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:43:41.734147 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080\": container with ID starting with cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080 not found: ID does not exist" containerID="cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080" Jul 10 13:43:41.734233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.734171 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080"} err="failed to get container status \"cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080\": rpc error: code = NotFound desc = could not find container \"cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080\": container with ID starting with cbe399b32c233035e4cb9ba6ce3c94e112aa6ba7e9b694ea389f2cd76dcfe080 not found: ID does not exist" Jul 10 13:43:41.796038 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796012 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-data\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796163 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796045 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-podinfo\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796163 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796074 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brs6j\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-kube-api-access-brs6j\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796163 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796111 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-token\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796295 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796163 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-certs\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796349 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796303 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-credential-socket\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796398 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796352 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51607bc-1d6a-4d88-a350-75f2caf259b0-istiod-ca-cert\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796449 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796434 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-envoy\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796470 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-socket\") pod \"a51607bc-1d6a-4d88-a350-75f2caf259b0\" (UID: \"a51607bc-1d6a-4d88-a350-75f2caf259b0\") " Jul 10 13:43:41.796554 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796499 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:41.796730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796706 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.796730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796715 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:41.796944 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796922 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:41.797016 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796974 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:41.797016 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.796982 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a51607bc-1d6a-4d88-a350-75f2caf259b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:43:41.798704 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.798676 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:43:41.798808 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.798708 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:43:41.798808 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.798785 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-kube-api-access-brs6j" (OuterVolumeSpecName: "kube-api-access-brs6j") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "kube-api-access-brs6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:43:41.798808 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.798795 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "a51607bc-1d6a-4d88-a350-75f2caf259b0" (UID: "a51607bc-1d6a-4d88-a350-75f2caf259b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:43:41.897368 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897345 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897368 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897366 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897375 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897384 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brs6j\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-kube-api-access-brs6j\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897393 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897400 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897408 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a51607bc-1d6a-4d88-a350-75f2caf259b0-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:41.897481 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:41.897416 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a51607bc-1d6a-4d88-a350-75f2caf259b0-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:43:42.032730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:42.032672 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg"] Jul 10 13:43:42.037083 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:42.037059 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-faacd6-openshift-default-7cdfbd586f-f2srg"] Jul 10 13:43:42.787771 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:42.787736 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:42.888223 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:42.888186 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:42.986947 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:42.986915 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:43.150010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:43.149910 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:43.249575 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:43.249539 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:43.405064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:43.404967 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:43.506728 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:43.506700 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:43.950920 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:43.950890 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" path="/var/lib/kubelet/pods/a51607bc-1d6a-4d88-a350-75f2caf259b0/volumes" Jul 10 13:43:47.338918 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:47.338881 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:47.844623 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:47.844585 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:48.148582 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.148498 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d"] Jul 10 13:43:48.148953 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.148912 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.148955 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.148971 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" containerName="main" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.148980 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" containerName="main" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149034 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149045 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149053 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149062 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" Jul 10 13:43:48.149073 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149075 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" Jul 10 13:43:48.149452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149084 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" Jul 10 13:43:48.149452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149200 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" Jul 10 13:43:48.149452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149211 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ba50b9-6da6-44ef-bb8e-1bb604bc59dc" containerName="main" Jul 10 13:43:48.149452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149223 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" Jul 10 13:43:48.149452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149232 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6454d6a-f4df-4052-a1fb-021223bbe30b" containerName="istio-proxy" Jul 10 13:43:48.149452 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.149244 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51607bc-1d6a-4d88-a350-75f2caf259b0" containerName="istio-proxy" Jul 10 13:43:48.153700 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.153680 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.156217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.156196 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e693af-openshift-default-dockercfg-7d984\"" Jul 10 13:43:48.163275 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.163256 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d"] Jul 10 13:43:48.185246 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.185218 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:48.255959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.255924 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-data\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256138 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256022 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvqt\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-kube-api-access-xjvqt\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256138 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-envoy\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256138 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256109 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5131a4c1-1755-48b9-9042-64813532a1f4-istio-podinfo\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256297 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256153 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-socket\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256297 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256174 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-certs\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256297 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256194 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-istio-token\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256297 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256281 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5131a4c1-1755-48b9-9042-64813532a1f4-istiod-ca-cert\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.256502 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.256387 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-credential-socket\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.357750 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.357716 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvqt\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-kube-api-access-xjvqt\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.357750 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.357752 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-envoy\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.357770 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5131a4c1-1755-48b9-9042-64813532a1f4-istio-podinfo\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.357893 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-socket\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.357926 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-certs\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.357942 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-istio-token\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358035 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5131a4c1-1755-48b9-9042-64813532a1f4-istiod-ca-cert\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358107 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-credential-socket\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358167 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-data\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358263 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-certs\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358418 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-credential-socket\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358612 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358531 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-data\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358807 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358781 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5131a4c1-1755-48b9-9042-64813532a1f4-istiod-ca-cert\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.358934 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.358911 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-socket\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.359994 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.359973 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-envoy\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.360222 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.360206 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5131a4c1-1755-48b9-9042-64813532a1f4-istio-podinfo\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.368061 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.368038 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-istio-token\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.368619 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.368596 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvqt\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-kube-api-access-xjvqt\") pod \"e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.466932 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.466905 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:48.545593 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.545564 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:48.603473 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.603447 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d"] Jul 10 13:43:48.605263 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:43:48.605236 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5131a4c1_1755_48b9_9042_64813532a1f4.slice/crio-9d372a143667a04e78e9de126d090ad7d6d42c0d7722af15bcc573086d3eb2af WatchSource:0}: Error finding container 9d372a143667a04e78e9de126d090ad7d6d42c0d7722af15bcc573086d3eb2af: Status 404 returned error can't find the container with id 9d372a143667a04e78e9de126d090ad7d6d42c0d7722af15bcc573086d3eb2af Jul 10 13:43:48.607388 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.607358 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:43:48.607458 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.607425 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:43:48.607501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.607457 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:43:48.757839 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.757755 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" event={"ID":"5131a4c1-1755-48b9-9042-64813532a1f4","Type":"ContainerStarted","Data":"6da51f89f483d6b0d94dfd6a9ac45d1d56f75936b2d8800c1503e7f25854325c"} Jul 10 13:43:48.757839 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.757813 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" event={"ID":"5131a4c1-1755-48b9-9042-64813532a1f4","Type":"ContainerStarted","Data":"9d372a143667a04e78e9de126d090ad7d6d42c0d7722af15bcc573086d3eb2af"} Jul 10 13:43:48.780373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:48.780308 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" podStartSLOduration=0.780287629 podStartE2EDuration="780.287629ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:43:48.777099082 +0000 UTC m=+2009.493476728" watchObservedRunningTime="2026-07-10 13:43:48.780287629 +0000 UTC m=+2009.496665257" Jul 10 13:43:49.468105 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:49.468071 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:49.473348 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:49.473323 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:49.764033 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:49.763963 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:49.764740 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:49.764721 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:43:50.767218 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:50.767152 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:50.853724 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:50.853693 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:50.925888 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:50.925857 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:51.152169 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:51.152093 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:51.520154 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:51.520117 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:51.621508 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:51.621475 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:43:55.312059 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:43:55.312020 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:03.728497 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.728455 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:03.833898 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.833871 2567 generic.go:356] "Generic (PLEG): container finished" podID="5131a4c1-1755-48b9-9042-64813532a1f4" containerID="6da51f89f483d6b0d94dfd6a9ac45d1d56f75936b2d8800c1503e7f25854325c" exitCode=0 Jul 10 13:44:03.834045 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.833912 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" event={"ID":"5131a4c1-1755-48b9-9042-64813532a1f4","Type":"ContainerDied","Data":"6da51f89f483d6b0d94dfd6a9ac45d1d56f75936b2d8800c1503e7f25854325c"} Jul 10 13:44:03.834339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.834313 2567 scope.go:117] "RemoveContainer" containerID="6da51f89f483d6b0d94dfd6a9ac45d1d56f75936b2d8800c1503e7f25854325c" Jul 10 13:44:03.835303 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.835275 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:44:03.835362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.835345 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:44:03.835401 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:03.835378 2567 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 13:44:04.108394 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:04.108292 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:04.323269 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:04.323235 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:04.421467 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:04.421386 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:04.675014 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:04.674919 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:04.764062 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:04.763969 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:04.844621 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:04.844582 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" event={"ID":"5131a4c1-1755-48b9-9042-64813532a1f4","Type":"ContainerStarted","Data":"d59ed07ce09c985a3192a59ea6fd9d6451a677851e10f3e48859e47fdeb3b81e"} Jul 10 13:44:05.467262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:05.467223 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:05.468282 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:05.468256 2567 patch_prober.go:28] interesting pod/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 10 13:44:05.468395 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:05.468306 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 10 13:44:06.467632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:06.467598 2567 patch_prober.go:28] interesting pod/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 10 13:44:06.468032 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:06.467647 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 10 13:44:07.468064 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:07.468028 2567 patch_prober.go:28] interesting pod/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 10 13:44:07.468443 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:07.468086 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 10 13:44:08.467823 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:08.467785 2567 patch_prober.go:28] interesting pod/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 10 13:44:08.468015 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:08.467836 2567 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 10 13:44:09.471048 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:09.471019 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:09.471463 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:09.471440 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:09.471998 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:09.471981 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:10.195632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:10.195594 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:10.298417 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:10.298379 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:10.465979 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:10.465901 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:10.717367 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:10.717275 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:10.881293 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:10.881257 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:11.283243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:11.283192 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:11.450922 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:11.450878 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:11.553748 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:11.553660 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:11.746809 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:11.746774 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:12.842723 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:12.842681 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:17.471409 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:17.471369 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:17.562020 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:17.561987 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:17.703636 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:17.703608 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d"] Jul 10 13:44:17.703836 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:17.703816 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" containerID="cri-o://d59ed07ce09c985a3192a59ea6fd9d6451a677851e10f3e48859e47fdeb3b81e" gracePeriod=30 Jul 10 13:44:17.729361 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:17.729285 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:17.930641 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:17.930604 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:18.251604 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:18.251564 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:18.563394 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:18.563298 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:19.167466 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:19.167437 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:21.862501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:21.862448 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:21.970241 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:21.970202 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:22.072258 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.072225 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:22.167714 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.167610 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:22.640536 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.640504 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:22.742698 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.742659 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:22.947148 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.947121 2567 generic.go:356] "Generic (PLEG): container finished" podID="5131a4c1-1755-48b9-9042-64813532a1f4" containerID="d59ed07ce09c985a3192a59ea6fd9d6451a677851e10f3e48859e47fdeb3b81e" exitCode=0 Jul 10 13:44:22.947449 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.947182 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" event={"ID":"5131a4c1-1755-48b9-9042-64813532a1f4","Type":"ContainerDied","Data":"d59ed07ce09c985a3192a59ea6fd9d6451a677851e10f3e48859e47fdeb3b81e"} Jul 10 13:44:22.947449 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.947266 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" event={"ID":"5131a4c1-1755-48b9-9042-64813532a1f4","Type":"ContainerDied","Data":"9d372a143667a04e78e9de126d090ad7d6d42c0d7722af15bcc573086d3eb2af"} Jul 10 13:44:22.947449 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.947286 2567 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d372a143667a04e78e9de126d090ad7d6d42c0d7722af15bcc573086d3eb2af" Jul 10 13:44:22.947449 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.947300 2567 scope.go:117] "RemoveContainer" containerID="6da51f89f483d6b0d94dfd6a9ac45d1d56f75936b2d8800c1503e7f25854325c" Jul 10 13:44:22.958129 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:22.958110 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:23.066299 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066274 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-certs\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066330 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjvqt\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-kube-api-access-xjvqt\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066351 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5131a4c1-1755-48b9-9042-64813532a1f4-istio-podinfo\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066515 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066486 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-credential-socket\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066573 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066546 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:23.066573 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066568 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5131a4c1-1755-48b9-9042-64813532a1f4-istiod-ca-cert\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066653 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066608 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-istio-token\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066653 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066639 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-data\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066758 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066671 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-envoy\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066758 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066701 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-socket\") pod \"5131a4c1-1755-48b9-9042-64813532a1f4\" (UID: \"5131a4c1-1755-48b9-9042-64813532a1f4\") " Jul 10 13:44:23.066758 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066720 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:23.067041 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.067017 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:23.067139 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.066952 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5131a4c1-1755-48b9-9042-64813532a1f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 13:44:23.067139 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.067092 2567 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.067139 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.067121 2567 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-credential-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.067904 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.067877 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:23.068662 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.068630 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5131a4c1-1755-48b9-9042-64813532a1f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 13:44:23.068762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.068702 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:44:23.068762 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.068741 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-kube-api-access-xjvqt" (OuterVolumeSpecName: "kube-api-access-xjvqt") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "kube-api-access-xjvqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:44:23.068959 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.068807 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5131a4c1-1755-48b9-9042-64813532a1f4" (UID: "5131a4c1-1755-48b9-9042-64813532a1f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:44:23.168251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168224 2567 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-data\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.168251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168247 2567 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-istio-envoy\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.168251 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168255 2567 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5131a4c1-1755-48b9-9042-64813532a1f4-workload-socket\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.168415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168264 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvqt\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-kube-api-access-xjvqt\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.168415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168274 2567 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5131a4c1-1755-48b9-9042-64813532a1f4-istio-podinfo\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.168415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168282 2567 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5131a4c1-1755-48b9-9042-64813532a1f4-istiod-ca-cert\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.168415 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.168289 2567 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5131a4c1-1755-48b9-9042-64813532a1f4-istio-token\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:44:23.953431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.953401 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d" Jul 10 13:44:23.985966 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.985941 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d"] Jul 10 13:44:23.992629 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:23.992605 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e693af-openshift-default-678cc7987c-wtn7d"] Jul 10 13:44:25.950254 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:25.950219 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" path="/var/lib/kubelet/pods/5131a4c1-1755-48b9-9042-64813532a1f4/volumes" Jul 10 13:44:26.382531 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:26.382443 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:26.725545 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:26.725510 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:27.091582 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:27.091504 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:27.645287 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:27.645250 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:29.971464 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:29.971426 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:30.068199 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:30.068164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:30.222615 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:30.222530 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:30.356615 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:30.356574 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:30.780752 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:30.780715 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:30.877208 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:30.877176 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:34.525683 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:34.525635 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:44.568317 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:44.568228 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:44.663404 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:44.663378 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:52.652478 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:52.652433 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:44:52.766192 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:44:52.766164 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:45:00.150725 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.150692 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z"] Jul 10 13:45:00.151200 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.151052 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" Jul 10 13:45:00.151200 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.151065 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" Jul 10 13:45:00.151200 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.151073 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" Jul 10 13:45:00.151200 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.151079 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" Jul 10 13:45:00.151200 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.151149 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" Jul 10 13:45:00.151200 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.151159 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="5131a4c1-1755-48b9-9042-64813532a1f4" containerName="istio-proxy" Jul 10 13:45:00.154003 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.153975 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" Jul 10 13:45:00.156028 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.155992 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz"] Jul 10 13:45:00.156928 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.156911 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hj9h2\"" Jul 10 13:45:00.159187 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.159171 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg"] Jul 10 13:45:00.159305 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.159290 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" Jul 10 13:45:00.162271 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.162247 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" Jul 10 13:45:00.165139 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.165119 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z"] Jul 10 13:45:00.166964 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.166337 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" Jul 10 13:45:00.169387 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.169366 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz"] Jul 10 13:45:00.178529 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.178510 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" Jul 10 13:45:00.187391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.187366 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg"] Jul 10 13:45:00.212327 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.212291 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" Jul 10 13:45:00.335312 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.335066 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z"] Jul 10 13:45:00.335721 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:45:00.335668 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f22b4d_314e_4073_b0f8_124fe5620db1.slice/crio-a52259aefd68ffe753fe91048c7f1f424f961f2d2db2e3dc78935e66cc2696fe WatchSource:0}: Error finding container a52259aefd68ffe753fe91048c7f1f424f961f2d2db2e3dc78935e66cc2696fe: Status 404 returned error can't find the container with id a52259aefd68ffe753fe91048c7f1f424f961f2d2db2e3dc78935e66cc2696fe Jul 10 13:45:00.337747 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.337718 2567 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 13:45:00.355953 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.355932 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz"] Jul 10 13:45:00.357767 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:45:00.357741 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3a955b_5a00_412d_8f6b_9ada8b7506fb.slice/crio-f401814efedc572725c4bf4febcd465a4e64a614171b9a5f260099c1aa65dff5 WatchSource:0}: Error finding container f401814efedc572725c4bf4febcd465a4e64a614171b9a5f260099c1aa65dff5: Status 404 returned error can't find the container with id f401814efedc572725c4bf4febcd465a4e64a614171b9a5f260099c1aa65dff5 Jul 10 13:45:00.382146 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:00.382127 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg"] Jul 10 13:45:00.384066 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:45:00.384044 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36da81a4_71e6_42d4_8612_d903825ae100.slice/crio-8c3f76c6bac81ad8493473cfe3a5e12d3f934ed6af39705dc3d9a62776027a74 WatchSource:0}: Error finding container 8c3f76c6bac81ad8493473cfe3a5e12d3f934ed6af39705dc3d9a62776027a74: Status 404 returned error can't find the container with id 8c3f76c6bac81ad8493473cfe3a5e12d3f934ed6af39705dc3d9a62776027a74 Jul 10 13:45:01.142431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.142397 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" event={"ID":"91f22b4d-314e-4073-b0f8-124fe5620db1","Type":"ContainerStarted","Data":"a52259aefd68ffe753fe91048c7f1f424f961f2d2db2e3dc78935e66cc2696fe"} Jul 10 13:45:01.143442 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.143419 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" event={"ID":"36da81a4-71e6-42d4-8612-d903825ae100","Type":"ContainerStarted","Data":"8c3f76c6bac81ad8493473cfe3a5e12d3f934ed6af39705dc3d9a62776027a74"} Jul 10 13:45:01.144454 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.144430 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" event={"ID":"ca3a955b-5a00-412d-8f6b-9ada8b7506fb","Type":"ContainerStarted","Data":"f401814efedc572725c4bf4febcd465a4e64a614171b9a5f260099c1aa65dff5"} Jul 10 13:45:01.931419 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.931385 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb"] Jul 10 13:45:01.936179 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.936153 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:01.939052 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.939030 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"kube-root-ca.crt\"" Jul 10 13:45:01.947388 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.940331 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"default-dockercfg-djsql\"" Jul 10 13:45:01.947388 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.940607 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"auth-test-model-90a164-kserve-self-signed-certs\"" Jul 10 13:45:01.947388 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.940700 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"openshift-service-ca.crt\"" Jul 10 13:45:01.956686 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.956664 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb"] Jul 10 13:45:01.965144 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:01.965067 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:45:02.072273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.071727 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-model-cache\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.072273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.071901 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-tmp-dir\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.072273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.071952 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-dshm\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.072273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.072081 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnxv\" (UniqueName: \"kubernetes.io/projected/e515e976-65a6-4772-b7cd-6e7092a751d2-kube-api-access-zwnxv\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.072273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.072233 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e515e976-65a6-4772-b7cd-6e7092a751d2-tls-certs\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.072647 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.072284 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-home\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.173912 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.173865 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnxv\" (UniqueName: \"kubernetes.io/projected/e515e976-65a6-4772-b7cd-6e7092a751d2-kube-api-access-zwnxv\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.173920 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e515e976-65a6-4772-b7cd-6e7092a751d2-tls-certs\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.173951 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-home\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.174003 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-model-cache\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.174054 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-tmp-dir\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174092 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.174088 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-dshm\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174530 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.174384 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-home\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174530 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.174430 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-model-cache\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.174530 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.174455 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-tmp-dir\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.177133 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.177109 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e515e976-65a6-4772-b7cd-6e7092a751d2-tls-certs\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.178078 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.178039 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-dshm\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.186099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.186026 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnxv\" (UniqueName: \"kubernetes.io/projected/e515e976-65a6-4772-b7cd-6e7092a751d2-kube-api-access-zwnxv\") pod \"auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.261837 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.261806 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:02.975839 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:02.975813 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb"] Jul 10 13:45:02.976943 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:45:02.976918 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode515e976_65a6_4772_b7cd_6e7092a751d2.slice/crio-142bbfb1315e2104d5d8f14a1455e75bfc3ae348874ceafbdc95fce77144e74e WatchSource:0}: Error finding container 142bbfb1315e2104d5d8f14a1455e75bfc3ae348874ceafbdc95fce77144e74e: Status 404 returned error can't find the container with id 142bbfb1315e2104d5d8f14a1455e75bfc3ae348874ceafbdc95fce77144e74e Jul 10 13:45:03.158094 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:03.158062 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" event={"ID":"e515e976-65a6-4772-b7cd-6e7092a751d2","Type":"ContainerStarted","Data":"6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e"} Jul 10 13:45:03.158206 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:03.158104 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" event={"ID":"e515e976-65a6-4772-b7cd-6e7092a751d2","Type":"ContainerStarted","Data":"142bbfb1315e2104d5d8f14a1455e75bfc3ae348874ceafbdc95fce77144e74e"} Jul 10 13:45:03.175880 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:03.175818 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" podStartSLOduration=2.175800808 podStartE2EDuration="2.175800808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:45:03.175461166 +0000 UTC m=+2083.891838847" watchObservedRunningTime="2026-07-10 13:45:03.175800808 +0000 UTC m=+2083.892178434" Jul 10 13:45:04.164514 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:04.164477 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" event={"ID":"36da81a4-71e6-42d4-8612-d903825ae100","Type":"ContainerStarted","Data":"4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded"} Jul 10 13:45:04.166029 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:04.166003 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" event={"ID":"ca3a955b-5a00-412d-8f6b-9ada8b7506fb","Type":"ContainerStarted","Data":"2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a"} Jul 10 13:45:04.167372 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:04.167350 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" event={"ID":"91f22b4d-314e-4073-b0f8-124fe5620db1","Type":"ContainerStarted","Data":"40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782"} Jul 10 13:45:04.181000 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:04.180959 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" podStartSLOduration=1.588095056 podStartE2EDuration="4.180947224s" totalImagesPullingTime="2.592852168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:45:04.17891887 +0000 UTC m=+2084.895296511" watchObservedRunningTime="2026-07-10 13:45:04.180947224 +0000 UTC m=+2084.897324849" Jul 10 13:45:04.193544 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:04.193503 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" podStartSLOduration=1.615815437 podStartE2EDuration="4.193489982s" totalImagesPullingTime="2.577674545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:45:04.192416648 +0000 UTC m=+2084.908794277" watchObservedRunningTime="2026-07-10 13:45:04.193489982 +0000 UTC m=+2084.909867607" Jul 10 13:45:04.207621 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:04.207586 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" podStartSLOduration=1.5947040970000002 podStartE2EDuration="4.207575553s" totalImagesPullingTime="2.612871456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:45:04.206792652 +0000 UTC m=+2084.923170279" watchObservedRunningTime="2026-07-10 13:45:04.207575553 +0000 UTC m=+2084.923953226" Jul 10 13:45:12.262914 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:12.262877 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:12.263333 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:12.262934 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:12.275856 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:12.275818 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:13.223691 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:13.223663 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:45:19.942777 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:19.942747 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:45:19.954924 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:19.954903 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:45:31.474775 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:31.474745 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:45:49.821037 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:45:49.820996 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:46:00.011035 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:00.011002 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg"] Jul 10 13:46:00.011503 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:00.011193 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" podUID="36da81a4-71e6-42d4-8612-d903825ae100" containerName="cleanup" containerID="cri-o://4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded" gracePeriod=30 Jul 10 13:46:00.014632 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:00.014610 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z"] Jul 10 13:46:00.014778 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:00.014761 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" podUID="91f22b4d-314e-4073-b0f8-124fe5620db1" containerName="cleanup" containerID="cri-o://40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782" gracePeriod=30 Jul 10 13:46:00.016558 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:00.016540 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz"] Jul 10 13:46:00.016740 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:00.016708 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" podUID="ca3a955b-5a00-412d-8f6b-9ada8b7506fb" containerName="cleanup" containerID="cri-o://2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a" gracePeriod=30 Jul 10 13:46:30.218160 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.218131 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" Jul 10 13:46:30.222419 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.222397 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" Jul 10 13:46:30.244339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.244316 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" Jul 10 13:46:30.616099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.616066 2567 generic.go:356] "Generic (PLEG): container finished" podID="36da81a4-71e6-42d4-8612-d903825ae100" containerID="4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded" exitCode=137 Jul 10 13:46:30.616272 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.616108 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" Jul 10 13:46:30.616272 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.616153 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" event={"ID":"36da81a4-71e6-42d4-8612-d903825ae100","Type":"ContainerDied","Data":"4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded"} Jul 10 13:46:30.616272 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.616215 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg" event={"ID":"36da81a4-71e6-42d4-8612-d903825ae100","Type":"ContainerDied","Data":"8c3f76c6bac81ad8493473cfe3a5e12d3f934ed6af39705dc3d9a62776027a74"} Jul 10 13:46:30.616272 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.616244 2567 scope.go:117] "RemoveContainer" containerID="4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded" Jul 10 13:46:30.617477 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.617450 2567 generic.go:356] "Generic (PLEG): container finished" podID="ca3a955b-5a00-412d-8f6b-9ada8b7506fb" containerID="2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a" exitCode=137 Jul 10 13:46:30.617605 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.617486 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" Jul 10 13:46:30.617605 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.617510 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" event={"ID":"ca3a955b-5a00-412d-8f6b-9ada8b7506fb","Type":"ContainerDied","Data":"2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a"} Jul 10 13:46:30.617605 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.617566 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz" event={"ID":"ca3a955b-5a00-412d-8f6b-9ada8b7506fb","Type":"ContainerDied","Data":"f401814efedc572725c4bf4febcd465a4e64a614171b9a5f260099c1aa65dff5"} Jul 10 13:46:30.619098 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.619069 2567 generic.go:356] "Generic (PLEG): container finished" podID="91f22b4d-314e-4073-b0f8-124fe5620db1" containerID="40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782" exitCode=137 Jul 10 13:46:30.619217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.619113 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" Jul 10 13:46:30.619217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.619133 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" event={"ID":"91f22b4d-314e-4073-b0f8-124fe5620db1","Type":"ContainerDied","Data":"40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782"} Jul 10 13:46:30.619217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.619173 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z" event={"ID":"91f22b4d-314e-4073-b0f8-124fe5620db1","Type":"ContainerDied","Data":"a52259aefd68ffe753fe91048c7f1f424f961f2d2db2e3dc78935e66cc2696fe"} Jul 10 13:46:30.626494 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.626361 2567 scope.go:117] "RemoveContainer" containerID="4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded" Jul 10 13:46:30.626824 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:46:30.626780 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded\": container with ID starting with 4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded not found: ID does not exist" containerID="4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded" Jul 10 13:46:30.626937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.626837 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded"} err="failed to get container status \"4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded\": rpc error: code = NotFound desc = could not find container \"4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded\": container with ID starting with 4def4fe6e2254f6a57aa9c1abfcec3c4de85d98ed4cd3d97dd55821a9fd91ded not found: ID does not exist" Jul 10 13:46:30.626937 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.626882 2567 scope.go:117] "RemoveContainer" containerID="2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a" Jul 10 13:46:30.634743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.634727 2567 scope.go:117] "RemoveContainer" containerID="2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a" Jul 10 13:46:30.635048 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:46:30.635028 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a\": container with ID starting with 2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a not found: ID does not exist" containerID="2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a" Jul 10 13:46:30.635133 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.635059 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a"} err="failed to get container status \"2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a\": rpc error: code = NotFound desc = could not find container \"2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a\": container with ID starting with 2faf6b823ff055f7f9199225537b3dab8fde046686fc9705edd73c72f09b235a not found: ID does not exist" Jul 10 13:46:30.635133 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.635079 2567 scope.go:117] "RemoveContainer" containerID="40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782" Jul 10 13:46:30.640765 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.640742 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg"] Jul 10 13:46:30.643291 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.643275 2567 scope.go:117] "RemoveContainer" containerID="40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782" Jul 10 13:46:30.643553 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:46:30.643536 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782\": container with ID starting with 40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782 not found: ID does not exist" containerID="40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782" Jul 10 13:46:30.643615 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.643558 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782"} err="failed to get container status \"40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782\": rpc error: code = NotFound desc = could not find container \"40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782\": container with ID starting with 40f7d665a9fc1b704c2ae080cf51eb930a47bd03e0defe38931ef790acadf782 not found: ID does not exist" Jul 10 13:46:30.644670 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.644651 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728185-p2bmg"] Jul 10 13:46:30.657099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.657079 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz"] Jul 10 13:46:30.661733 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.661714 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-90a164-29728185-j8qfz"] Jul 10 13:46:30.671548 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.671529 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z"] Jul 10 13:46:30.675386 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:30.675364 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-efd2ad-29728185-q678z"] Jul 10 13:46:31.950519 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:31.950485 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36da81a4-71e6-42d4-8612-d903825ae100" path="/var/lib/kubelet/pods/36da81a4-71e6-42d4-8612-d903825ae100/volumes" Jul 10 13:46:31.951008 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:31.950865 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f22b4d-314e-4073-b0f8-124fe5620db1" path="/var/lib/kubelet/pods/91f22b4d-314e-4073-b0f8-124fe5620db1/volumes" Jul 10 13:46:31.951176 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:46:31.951156 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca3a955b-5a00-412d-8f6b-9ada8b7506fb" path="/var/lib/kubelet/pods/ca3a955b-5a00-412d-8f6b-9ada8b7506fb/volumes" Jul 10 13:47:08.674555 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:08.674519 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:47:08.727925 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:08.727892 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:47:38.916962 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:38.916928 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb"] Jul 10 13:47:38.939942 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:38.939907 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:47:39.755990 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:39.755955 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:47:53.936753 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:53.936650 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" podUID="e515e976-65a6-4772-b7cd-6e7092a751d2" containerName="main" containerID="cri-o://6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e" gracePeriod=45 Jul 10 13:47:54.186393 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.186367 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:47:54.294498 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294475 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-tmp-dir\") pod \"e515e976-65a6-4772-b7cd-6e7092a751d2\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " Jul 10 13:47:54.294678 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294526 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-dshm\") pod \"e515e976-65a6-4772-b7cd-6e7092a751d2\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " Jul 10 13:47:54.294678 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294565 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e515e976-65a6-4772-b7cd-6e7092a751d2-tls-certs\") pod \"e515e976-65a6-4772-b7cd-6e7092a751d2\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " Jul 10 13:47:54.294678 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294594 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-model-cache\") pod \"e515e976-65a6-4772-b7cd-6e7092a751d2\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " Jul 10 13:47:54.294678 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294609 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-home\") pod \"e515e976-65a6-4772-b7cd-6e7092a751d2\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " Jul 10 13:47:54.294678 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294630 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwnxv\" (UniqueName: \"kubernetes.io/projected/e515e976-65a6-4772-b7cd-6e7092a751d2-kube-api-access-zwnxv\") pod \"e515e976-65a6-4772-b7cd-6e7092a751d2\" (UID: \"e515e976-65a6-4772-b7cd-6e7092a751d2\") " Jul 10 13:47:54.294986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294754 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e515e976-65a6-4772-b7cd-6e7092a751d2" (UID: "e515e976-65a6-4772-b7cd-6e7092a751d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:47:54.294986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294901 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e515e976-65a6-4772-b7cd-6e7092a751d2" (UID: "e515e976-65a6-4772-b7cd-6e7092a751d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:47:54.294986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294917 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-home" (OuterVolumeSpecName: "home") pod "e515e976-65a6-4772-b7cd-6e7092a751d2" (UID: "e515e976-65a6-4772-b7cd-6e7092a751d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:47:54.294986 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.294967 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:47:54.296795 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.296763 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e515e976-65a6-4772-b7cd-6e7092a751d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e515e976-65a6-4772-b7cd-6e7092a751d2" (UID: "e515e976-65a6-4772-b7cd-6e7092a751d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:47:54.296944 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.296818 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-dshm" (OuterVolumeSpecName: "dshm") pod "e515e976-65a6-4772-b7cd-6e7092a751d2" (UID: "e515e976-65a6-4772-b7cd-6e7092a751d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:47:54.296944 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.296857 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e515e976-65a6-4772-b7cd-6e7092a751d2-kube-api-access-zwnxv" (OuterVolumeSpecName: "kube-api-access-zwnxv") pod "e515e976-65a6-4772-b7cd-6e7092a751d2" (UID: "e515e976-65a6-4772-b7cd-6e7092a751d2"). InnerVolumeSpecName "kube-api-access-zwnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:47:54.395603 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.395578 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e515e976-65a6-4772-b7cd-6e7092a751d2-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:47:54.395603 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.395600 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:47:54.395739 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.395609 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:47:54.395739 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.395618 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnxv\" (UniqueName: \"kubernetes.io/projected/e515e976-65a6-4772-b7cd-6e7092a751d2-kube-api-access-zwnxv\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:47:54.395739 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:54.395627 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e515e976-65a6-4772-b7cd-6e7092a751d2-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:47:55.050362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.050328 2567 generic.go:356] "Generic (PLEG): container finished" podID="e515e976-65a6-4772-b7cd-6e7092a751d2" containerID="6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e" exitCode=0 Jul 10 13:47:55.050362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.050364 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" Jul 10 13:47:55.050890 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.050414 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" event={"ID":"e515e976-65a6-4772-b7cd-6e7092a751d2","Type":"ContainerDied","Data":"6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e"} Jul 10 13:47:55.050890 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.050451 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb" event={"ID":"e515e976-65a6-4772-b7cd-6e7092a751d2","Type":"ContainerDied","Data":"142bbfb1315e2104d5d8f14a1455e75bfc3ae348874ceafbdc95fce77144e74e"} Jul 10 13:47:55.050890 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.050469 2567 scope.go:117] "RemoveContainer" containerID="6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e" Jul 10 13:47:55.059186 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.059171 2567 scope.go:117] "RemoveContainer" containerID="6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e" Jul 10 13:47:55.059457 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:47:55.059438 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e\": container with ID starting with 6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e not found: ID does not exist" containerID="6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e" Jul 10 13:47:55.059506 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.059466 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e"} err="failed to get container status \"6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e\": rpc error: code = NotFound desc = could not find container \"6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e\": container with ID starting with 6d5ec4f810e8dad4286bbde688f1ef0cedb69bbc2d7f5cfdf6c83e5fe4916e2e not found: ID does not exist" Jul 10 13:47:55.071472 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.071452 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb"] Jul 10 13:47:55.076026 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.076006 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/auth-test-model-90a164-kserve-6b4f7d5bff-pxtvb"] Jul 10 13:47:55.950210 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:47:55.950179 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e515e976-65a6-4772-b7cd-6e7092a751d2" path="/var/lib/kubelet/pods/e515e976-65a6-4772-b7cd-6e7092a751d2/volumes" Jul 10 13:48:02.452367 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452324 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks"] Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452672 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e515e976-65a6-4772-b7cd-6e7092a751d2" containerName="main" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452683 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e515e976-65a6-4772-b7cd-6e7092a751d2" containerName="main" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452691 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca3a955b-5a00-412d-8f6b-9ada8b7506fb" containerName="cleanup" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452697 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca3a955b-5a00-412d-8f6b-9ada8b7506fb" containerName="cleanup" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452709 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36da81a4-71e6-42d4-8612-d903825ae100" containerName="cleanup" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452714 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="36da81a4-71e6-42d4-8612-d903825ae100" containerName="cleanup" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452723 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f22b4d-314e-4073-b0f8-124fe5620db1" containerName="cleanup" Jul 10 13:48:02.452761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452728 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f22b4d-314e-4073-b0f8-124fe5620db1" containerName="cleanup" Jul 10 13:48:02.453046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452784 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e515e976-65a6-4772-b7cd-6e7092a751d2" containerName="main" Jul 10 13:48:02.453046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452793 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f22b4d-314e-4073-b0f8-124fe5620db1" containerName="cleanup" Jul 10 13:48:02.453046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452801 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="36da81a4-71e6-42d4-8612-d903825ae100" containerName="cleanup" Jul 10 13:48:02.453046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.452806 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca3a955b-5a00-412d-8f6b-9ada8b7506fb" containerName="cleanup" Jul 10 13:48:02.456941 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.456920 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.459501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.459477 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"kube-root-ca.crt\"" Jul 10 13:48:02.460458 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.460435 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"sub-test-model-90a164-kserve-self-signed-certs\"" Jul 10 13:48:02.460591 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.460514 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"openshift-service-ca.crt\"" Jul 10 13:48:02.460591 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.460522 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"default-dockercfg-djsql\"" Jul 10 13:48:02.465051 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.465027 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks"] Jul 10 13:48:02.534983 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.534945 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:48:02.572998 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.572960 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-home\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.573128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.573004 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-tmp-dir\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.573128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.573028 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-model-cache\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.573218 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.573120 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de440ee1-c972-45a7-878e-c76ba70b77c1-tls-certs\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.573218 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.573152 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-dshm\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.573218 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.573171 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggb7s\" (UniqueName: \"kubernetes.io/projected/de440ee1-c972-45a7-878e-c76ba70b77c1-kube-api-access-ggb7s\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.673694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.673660 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-home\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.673694 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.673700 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-tmp-dir\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.673974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.673719 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-model-cache\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.673974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.673750 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de440ee1-c972-45a7-878e-c76ba70b77c1-tls-certs\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.673974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.673778 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-dshm\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.673974 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.673796 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggb7s\" (UniqueName: \"kubernetes.io/projected/de440ee1-c972-45a7-878e-c76ba70b77c1-kube-api-access-ggb7s\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.674165 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.674089 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-tmp-dir\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.674226 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.674162 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-home\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.674226 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.674195 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-model-cache\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.676213 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.676183 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-dshm\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.676353 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.676334 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de440ee1-c972-45a7-878e-c76ba70b77c1-tls-certs\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.684507 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.684482 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggb7s\" (UniqueName: \"kubernetes.io/projected/de440ee1-c972-45a7-878e-c76ba70b77c1-kube-api-access-ggb7s\") pod \"sub-test-model-90a164-kserve-7c7f7854cc-gs5ks\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.767273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.767193 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:02.895707 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:02.895682 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks"] Jul 10 13:48:02.898289 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:48:02.898261 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde440ee1_c972_45a7_878e_c76ba70b77c1.slice/crio-321408601c3dbca6d8835363514a5c50d15a05a4d73f973ac96686fb68a99b0a WatchSource:0}: Error finding container 321408601c3dbca6d8835363514a5c50d15a05a4d73f973ac96686fb68a99b0a: Status 404 returned error can't find the container with id 321408601c3dbca6d8835363514a5c50d15a05a4d73f973ac96686fb68a99b0a Jul 10 13:48:03.093820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:03.093724 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" event={"ID":"de440ee1-c972-45a7-878e-c76ba70b77c1","Type":"ContainerStarted","Data":"451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2"} Jul 10 13:48:03.093820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:03.093768 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" event={"ID":"de440ee1-c972-45a7-878e-c76ba70b77c1","Type":"ContainerStarted","Data":"321408601c3dbca6d8835363514a5c50d15a05a4d73f973ac96686fb68a99b0a"} Jul 10 13:48:03.111043 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:03.110989 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" podStartSLOduration=1.110973491 podStartE2EDuration="1.110973491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:48:03.109683535 +0000 UTC m=+2263.826061173" watchObservedRunningTime="2026-07-10 13:48:03.110973491 +0000 UTC m=+2263.827351118" Jul 10 13:48:12.767759 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:12.767723 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:12.768190 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:12.767770 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:12.780276 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:12.780251 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:13.154322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:13.154234 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:48:36.702753 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:36.702721 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:48:36.805046 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:36.805016 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:48:41.640368 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.640335 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9"] Jul 10 13:48:41.643936 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.643915 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.646659 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.646639 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"kube-root-ca.crt\"" Jul 10 13:48:41.646770 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.646705 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"openshift-service-ca.crt\"" Jul 10 13:48:41.647787 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.647767 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"sub-test-model-efd2ad-kserve-self-signed-certs\"" Jul 10 13:48:41.647912 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.647794 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"default-dockercfg-bnqhj\"" Jul 10 13:48:41.652202 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.652177 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9"] Jul 10 13:48:41.665574 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.665548 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:48:41.692684 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.692660 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-dshm\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.692778 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.692702 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tmp-dir\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.692778 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.692722 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-model-cache\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.692778 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.692746 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tls-certs\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.692907 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.692795 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5v6\" (UniqueName: \"kubernetes.io/projected/8e8bbd96-e4ec-4348-95ee-6349de8dd218-kube-api-access-6s5v6\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.692907 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.692815 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-home\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794019 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.793984 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tmp-dir\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794024 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-model-cache\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794065 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tls-certs\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794121 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5v6\" (UniqueName: \"kubernetes.io/projected/8e8bbd96-e4ec-4348-95ee-6349de8dd218-kube-api-access-6s5v6\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794189 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794151 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-home\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794406 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794213 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-dshm\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794483 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794442 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tmp-dir\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794543 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-model-cache\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.794594 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.794541 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-home\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.796314 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.796294 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-dshm\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.796496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.796479 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tls-certs\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.802220 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.802198 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5v6\" (UniqueName: \"kubernetes.io/projected/8e8bbd96-e4ec-4348-95ee-6349de8dd218-kube-api-access-6s5v6\") pod \"sub-test-model-efd2ad-kserve-7877976c59-v95c9\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:41.954322 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:41.954294 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:42.081758 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:42.081734 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9"] Jul 10 13:48:42.083086 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:48:42.083058 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e8bbd96_e4ec_4348_95ee_6349de8dd218.slice/crio-c84c14fd087f6004fc156aeeae73528ed95c6cd7b27d5568dc5b136a0aed24b2 WatchSource:0}: Error finding container c84c14fd087f6004fc156aeeae73528ed95c6cd7b27d5568dc5b136a0aed24b2: Status 404 returned error can't find the container with id c84c14fd087f6004fc156aeeae73528ed95c6cd7b27d5568dc5b136a0aed24b2 Jul 10 13:48:42.293739 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:42.293429 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" event={"ID":"8e8bbd96-e4ec-4348-95ee-6349de8dd218","Type":"ContainerStarted","Data":"6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272"} Jul 10 13:48:42.293739 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:42.293482 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" event={"ID":"8e8bbd96-e4ec-4348-95ee-6349de8dd218","Type":"ContainerStarted","Data":"c84c14fd087f6004fc156aeeae73528ed95c6cd7b27d5568dc5b136a0aed24b2"} Jul 10 13:48:42.312034 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:42.311982 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" podStartSLOduration=1.311965605 podStartE2EDuration="1.311965605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:48:42.309390137 +0000 UTC m=+2303.025767782" watchObservedRunningTime="2026-07-10 13:48:42.311965605 +0000 UTC m=+2303.028343230" Jul 10 13:48:51.954736 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:51.954705 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:51.955167 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:51.954752 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:51.966925 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:51.966904 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:48:52.355260 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:48:52.355177 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:49:18.438522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:18.438431 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:27.227808 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:27.227770 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks"] Jul 10 13:49:27.241236 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:27.241203 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:28.037645 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:28.037568 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9"] Jul 10 13:49:28.058833 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:28.058793 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:34.333788 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.333753 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth"] Jul 10 13:49:34.337554 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.337532 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.340225 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.340203 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-90a164\"/\"rate-test-model-90a164-kserve-self-signed-certs\"" Jul 10 13:49:34.358496 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.358473 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth"] Jul 10 13:49:34.377476 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.377439 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:34.429378 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.429351 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-dshm\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.429484 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.429392 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tls-certs\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.429484 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.429418 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-489ll\" (UniqueName: \"kubernetes.io/projected/172b992f-bc76-473f-a9e3-ae00bc1fcb66-kube-api-access-489ll\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.429592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.429490 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-model-cache\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.429592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.429520 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tmp-dir\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.429592 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.429538 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-home\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530243 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530208 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-489ll\" (UniqueName: \"kubernetes.io/projected/172b992f-bc76-473f-a9e3-ae00bc1fcb66-kube-api-access-489ll\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530430 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530274 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-model-cache\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530430 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530317 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tmp-dir\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530430 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530343 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-home\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530430 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530401 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-dshm\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530664 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530453 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tls-certs\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530726 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530698 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-model-cache\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530770 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530743 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tmp-dir\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.530806 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.530754 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-home\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.532642 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.532618 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-dshm\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.533031 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.533011 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tls-certs\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.538420 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.538399 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-489ll\" (UniqueName: \"kubernetes.io/projected/172b992f-bc76-473f-a9e3-ae00bc1fcb66-kube-api-access-489ll\") pod \"rate-test-model-90a164-kserve-7476498f75-4gdth\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.646291 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.646237 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:34.778087 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:34.778052 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth"] Jul 10 13:49:34.778600 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:49:34.778574 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod172b992f_bc76_473f_a9e3_ae00bc1fcb66.slice/crio-92a2cc5fb974ed72133fd0fe93cf6440022bfe56685682bfb2ad96a7c7a90eae WatchSource:0}: Error finding container 92a2cc5fb974ed72133fd0fe93cf6440022bfe56685682bfb2ad96a7c7a90eae: Status 404 returned error can't find the container with id 92a2cc5fb974ed72133fd0fe93cf6440022bfe56685682bfb2ad96a7c7a90eae Jul 10 13:49:35.563743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:35.563705 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" event={"ID":"172b992f-bc76-473f-a9e3-ae00bc1fcb66","Type":"ContainerStarted","Data":"8ab9e90b25b8143d54504f03cc61e03ada4f98b3af03e22ba001ea11eb453d36"} Jul 10 13:49:35.564126 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:35.563751 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" event={"ID":"172b992f-bc76-473f-a9e3-ae00bc1fcb66","Type":"ContainerStarted","Data":"92a2cc5fb974ed72133fd0fe93cf6440022bfe56685682bfb2ad96a7c7a90eae"} Jul 10 13:49:35.581801 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:35.581755 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" podStartSLOduration=1.581740529 podStartE2EDuration="1.581740529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:49:35.580165751 +0000 UTC m=+2356.296543402" watchObservedRunningTime="2026-07-10 13:49:35.581740529 +0000 UTC m=+2356.298118155" Jul 10 13:49:42.542534 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.542496 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" podUID="de440ee1-c972-45a7-878e-c76ba70b77c1" containerName="main" containerID="cri-o://451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2" gracePeriod=45 Jul 10 13:49:42.790890 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.790868 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:49:42.902061 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.901993 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-model-cache\") pod \"de440ee1-c972-45a7-878e-c76ba70b77c1\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " Jul 10 13:49:42.902061 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902047 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-dshm\") pod \"de440ee1-c972-45a7-878e-c76ba70b77c1\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " Jul 10 13:49:42.902221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902081 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggb7s\" (UniqueName: \"kubernetes.io/projected/de440ee1-c972-45a7-878e-c76ba70b77c1-kube-api-access-ggb7s\") pod \"de440ee1-c972-45a7-878e-c76ba70b77c1\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " Jul 10 13:49:42.902221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902138 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de440ee1-c972-45a7-878e-c76ba70b77c1-tls-certs\") pod \"de440ee1-c972-45a7-878e-c76ba70b77c1\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " Jul 10 13:49:42.902221 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902182 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-tmp-dir\") pod \"de440ee1-c972-45a7-878e-c76ba70b77c1\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " Jul 10 13:49:42.902337 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902242 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-home\") pod \"de440ee1-c972-45a7-878e-c76ba70b77c1\" (UID: \"de440ee1-c972-45a7-878e-c76ba70b77c1\") " Jul 10 13:49:42.902337 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902299 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "de440ee1-c972-45a7-878e-c76ba70b77c1" (UID: "de440ee1-c972-45a7-878e-c76ba70b77c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:42.902507 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902461 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de440ee1-c972-45a7-878e-c76ba70b77c1" (UID: "de440ee1-c972-45a7-878e-c76ba70b77c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:42.902507 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902502 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:42.902648 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.902544 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-home" (OuterVolumeSpecName: "home") pod "de440ee1-c972-45a7-878e-c76ba70b77c1" (UID: "de440ee1-c972-45a7-878e-c76ba70b77c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:42.904247 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.904217 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-dshm" (OuterVolumeSpecName: "dshm") pod "de440ee1-c972-45a7-878e-c76ba70b77c1" (UID: "de440ee1-c972-45a7-878e-c76ba70b77c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:42.904339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.904256 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de440ee1-c972-45a7-878e-c76ba70b77c1-kube-api-access-ggb7s" (OuterVolumeSpecName: "kube-api-access-ggb7s") pod "de440ee1-c972-45a7-878e-c76ba70b77c1" (UID: "de440ee1-c972-45a7-878e-c76ba70b77c1"). InnerVolumeSpecName "kube-api-access-ggb7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:49:42.904339 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:42.904295 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de440ee1-c972-45a7-878e-c76ba70b77c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de440ee1-c972-45a7-878e-c76ba70b77c1" (UID: "de440ee1-c972-45a7-878e-c76ba70b77c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:49:43.003599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.003574 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.003599 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.003596 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggb7s\" (UniqueName: \"kubernetes.io/projected/de440ee1-c972-45a7-878e-c76ba70b77c1-kube-api-access-ggb7s\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.003719 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.003606 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de440ee1-c972-45a7-878e-c76ba70b77c1-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.003719 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.003615 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.003719 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.003623 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de440ee1-c972-45a7-878e-c76ba70b77c1-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.059120 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.059067 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" podUID="8e8bbd96-e4ec-4348-95ee-6349de8dd218" containerName="main" containerID="cri-o://6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272" gracePeriod=45 Jul 10 13:49:43.290696 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.290673 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:49:43.407538 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407506 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-dshm\") pod \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " Jul 10 13:49:43.407538 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407540 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tls-certs\") pod \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " Jul 10 13:49:43.407764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407593 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-model-cache\") pod \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " Jul 10 13:49:43.407764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407619 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-home\") pod \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " Jul 10 13:49:43.407764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407644 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tmp-dir\") pod \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " Jul 10 13:49:43.407764 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407693 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s5v6\" (UniqueName: \"kubernetes.io/projected/8e8bbd96-e4ec-4348-95ee-6349de8dd218-kube-api-access-6s5v6\") pod \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\" (UID: \"8e8bbd96-e4ec-4348-95ee-6349de8dd218\") " Jul 10 13:49:43.408002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407889 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-model-cache" (OuterVolumeSpecName: "model-cache") pod "8e8bbd96-e4ec-4348-95ee-6349de8dd218" (UID: "8e8bbd96-e4ec-4348-95ee-6349de8dd218"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:43.408002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407899 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8e8bbd96-e4ec-4348-95ee-6349de8dd218" (UID: "8e8bbd96-e4ec-4348-95ee-6349de8dd218"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:43.408002 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.407936 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-home" (OuterVolumeSpecName: "home") pod "8e8bbd96-e4ec-4348-95ee-6349de8dd218" (UID: "8e8bbd96-e4ec-4348-95ee-6349de8dd218"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:43.408148 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.408056 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.408148 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.408073 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.408148 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.408086 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.409756 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.409736 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8e8bbd96-e4ec-4348-95ee-6349de8dd218" (UID: "8e8bbd96-e4ec-4348-95ee-6349de8dd218"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:49:43.410051 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.410033 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-dshm" (OuterVolumeSpecName: "dshm") pod "8e8bbd96-e4ec-4348-95ee-6349de8dd218" (UID: "8e8bbd96-e4ec-4348-95ee-6349de8dd218"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:49:43.410130 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.410032 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8bbd96-e4ec-4348-95ee-6349de8dd218-kube-api-access-6s5v6" (OuterVolumeSpecName: "kube-api-access-6s5v6") pod "8e8bbd96-e4ec-4348-95ee-6349de8dd218" (UID: "8e8bbd96-e4ec-4348-95ee-6349de8dd218"). InnerVolumeSpecName "kube-api-access-6s5v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:49:43.508701 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.508677 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e8bbd96-e4ec-4348-95ee-6349de8dd218-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.508701 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.508697 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e8bbd96-e4ec-4348-95ee-6349de8dd218-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.508817 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.508708 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s5v6\" (UniqueName: \"kubernetes.io/projected/8e8bbd96-e4ec-4348-95ee-6349de8dd218-kube-api-access-6s5v6\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:49:43.604910 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.604879 2567 generic.go:356] "Generic (PLEG): container finished" podID="8e8bbd96-e4ec-4348-95ee-6349de8dd218" containerID="6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272" exitCode=0 Jul 10 13:49:43.605252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.604919 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" Jul 10 13:49:43.605252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.604967 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" event={"ID":"8e8bbd96-e4ec-4348-95ee-6349de8dd218","Type":"ContainerDied","Data":"6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272"} Jul 10 13:49:43.605252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.605009 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9" event={"ID":"8e8bbd96-e4ec-4348-95ee-6349de8dd218","Type":"ContainerDied","Data":"c84c14fd087f6004fc156aeeae73528ed95c6cd7b27d5568dc5b136a0aed24b2"} Jul 10 13:49:43.605252 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.605031 2567 scope.go:117] "RemoveContainer" containerID="6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272" Jul 10 13:49:43.606419 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.606391 2567 generic.go:356] "Generic (PLEG): container finished" podID="de440ee1-c972-45a7-878e-c76ba70b77c1" containerID="451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2" exitCode=0 Jul 10 13:49:43.606512 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.606423 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" event={"ID":"de440ee1-c972-45a7-878e-c76ba70b77c1","Type":"ContainerDied","Data":"451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2"} Jul 10 13:49:43.606512 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.606464 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" event={"ID":"de440ee1-c972-45a7-878e-c76ba70b77c1","Type":"ContainerDied","Data":"321408601c3dbca6d8835363514a5c50d15a05a4d73f973ac96686fb68a99b0a"} Jul 10 13:49:43.606512 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.606492 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks" Jul 10 13:49:43.614962 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.614943 2567 scope.go:117] "RemoveContainer" containerID="6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272" Jul 10 13:49:43.615225 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:49:43.615206 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272\": container with ID starting with 6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272 not found: ID does not exist" containerID="6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272" Jul 10 13:49:43.615300 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.615232 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272"} err="failed to get container status \"6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272\": rpc error: code = NotFound desc = could not find container \"6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272\": container with ID starting with 6a7efa64beab8b8e22ec93181c865f820149483e20283c4df13ab8633d61c272 not found: ID does not exist" Jul 10 13:49:43.615300 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.615249 2567 scope.go:117] "RemoveContainer" containerID="451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2" Jul 10 13:49:43.623067 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.623051 2567 scope.go:117] "RemoveContainer" containerID="451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2" Jul 10 13:49:43.623325 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:49:43.623307 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2\": container with ID starting with 451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2 not found: ID does not exist" containerID="451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2" Jul 10 13:49:43.623384 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.623331 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2"} err="failed to get container status \"451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2\": rpc error: code = NotFound desc = could not find container \"451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2\": container with ID starting with 451099d710977bd6aa7030804ceaddcf1f42523a8a2aa663c8b5a3af2fb445f2 not found: ID does not exist" Jul 10 13:49:43.628641 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.628620 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9"] Jul 10 13:49:43.632373 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.632349 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/sub-test-model-efd2ad-kserve-7877976c59-v95c9"] Jul 10 13:49:43.641650 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.641627 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks"] Jul 10 13:49:43.647335 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.647313 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/sub-test-model-90a164-kserve-7c7f7854cc-gs5ks"] Jul 10 13:49:43.950723 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.950695 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8bbd96-e4ec-4348-95ee-6349de8dd218" path="/var/lib/kubelet/pods/8e8bbd96-e4ec-4348-95ee-6349de8dd218/volumes" Jul 10 13:49:43.951145 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:43.951130 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de440ee1-c972-45a7-878e-c76ba70b77c1" path="/var/lib/kubelet/pods/de440ee1-c972-45a7-878e-c76ba70b77c1/volumes" Jul 10 13:49:44.647228 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:44.647187 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:44.647666 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:44.647241 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:44.660049 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:44.660022 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:45.421362 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:45.421324 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:45.522004 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:45.521974 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:45.630673 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:45.630641 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:49:53.491828 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:53.491790 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:49:53.591423 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:49:53.591388 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:02.638852 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.638804 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn"] Jul 10 13:50:02.639245 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.639208 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de440ee1-c972-45a7-878e-c76ba70b77c1" containerName="main" Jul 10 13:50:02.639245 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.639223 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="de440ee1-c972-45a7-878e-c76ba70b77c1" containerName="main" Jul 10 13:50:02.639245 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.639232 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8bbd96-e4ec-4348-95ee-6349de8dd218" containerName="main" Jul 10 13:50:02.639245 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.639237 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8bbd96-e4ec-4348-95ee-6349de8dd218" containerName="main" Jul 10 13:50:02.639391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.639322 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="de440ee1-c972-45a7-878e-c76ba70b77c1" containerName="main" Jul 10 13:50:02.639391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.639332 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8bbd96-e4ec-4348-95ee-6349de8dd218" containerName="main" Jul 10 13:50:02.642258 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.642243 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.645030 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.645006 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"openshift-service-ca.crt\"" Jul 10 13:50:02.645149 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.645037 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"default-dockercfg-bnqhj\"" Jul 10 13:50:02.646111 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.646087 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"kube-root-ca.crt\"" Jul 10 13:50:02.646229 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.646195 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-efd2ad\"/\"rate-test-model-efd2ad-kserve-self-signed-certs\"" Jul 10 13:50:02.654677 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.654656 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn"] Jul 10 13:50:02.679805 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.679774 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:50:02.760404 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.760380 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-home\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.760504 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.760415 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-tmp-dir\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.760504 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.760433 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-model-cache\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.760586 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.760505 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73b4b4e-1c42-4a05-a206-b11f6206786b-tls-certs\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.760586 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.760530 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqt2l\" (UniqueName: \"kubernetes.io/projected/e73b4b4e-1c42-4a05-a206-b11f6206786b-kube-api-access-sqt2l\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.760651 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.760600 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-dshm\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.861709 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.861680 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-dshm\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.861910 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.861720 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-home\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.861910 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.861745 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-tmp-dir\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.861910 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.861766 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-model-cache\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.861910 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.861809 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73b4b4e-1c42-4a05-a206-b11f6206786b-tls-certs\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.861910 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.861869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqt2l\" (UniqueName: \"kubernetes.io/projected/e73b4b4e-1c42-4a05-a206-b11f6206786b-kube-api-access-sqt2l\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.862198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.862159 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-model-cache\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.862262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.862207 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-tmp-dir\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.862262 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.862254 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-home\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.864122 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.864099 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-dshm\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.864222 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.864205 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73b4b4e-1c42-4a05-a206-b11f6206786b-tls-certs\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.870111 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.870092 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqt2l\" (UniqueName: \"kubernetes.io/projected/e73b4b4e-1c42-4a05-a206-b11f6206786b-kube-api-access-sqt2l\") pod \"rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:02.952807 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:02.952790 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:03.085591 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:03.085563 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn"] Jul 10 13:50:03.087353 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:50:03.087329 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73b4b4e_1c42_4a05_a206_b11f6206786b.slice/crio-1a4ac52e8fbaac8d9ec8d185dd29fa6e998329754dc9d8eec84bd692ec3cc784 WatchSource:0}: Error finding container 1a4ac52e8fbaac8d9ec8d185dd29fa6e998329754dc9d8eec84bd692ec3cc784: Status 404 returned error can't find the container with id 1a4ac52e8fbaac8d9ec8d185dd29fa6e998329754dc9d8eec84bd692ec3cc784 Jul 10 13:50:03.714621 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:03.714590 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" event={"ID":"e73b4b4e-1c42-4a05-a206-b11f6206786b","Type":"ContainerStarted","Data":"e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63"} Jul 10 13:50:03.715022 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:03.714629 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" event={"ID":"e73b4b4e-1c42-4a05-a206-b11f6206786b","Type":"ContainerStarted","Data":"1a4ac52e8fbaac8d9ec8d185dd29fa6e998329754dc9d8eec84bd692ec3cc784"} Jul 10 13:50:03.737096 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:03.737055 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" podStartSLOduration=1.737041179 podStartE2EDuration="1.737041179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 13:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:50:03.735254677 +0000 UTC m=+2384.451632305" watchObservedRunningTime="2026-07-10 13:50:03.737041179 +0000 UTC m=+2384.453418830" Jul 10 13:50:12.953735 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:12.953701 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:12.954301 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:12.953746 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:12.966127 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:12.966105 2567 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:13.776667 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:13.776635 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:50:19.982829 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:19.982791 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:50:19.989407 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:19.989383 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knf4x_09abb34b-0b90-4794-9ff3-87a3f23fed12/ovn-acl-logging/0.log" Jul 10 13:50:20.069731 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:20.069707 2567 scope.go:117] "RemoveContainer" containerID="d59ed07ce09c985a3192a59ea6fd9d6451a677851e10f3e48859e47fdeb3b81e" Jul 10 13:50:50.567500 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:50:50.567409 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:12.986552 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:12.986521 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth"] Jul 10 13:51:13.004446 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:13.004414 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:13.755521 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:13.755490 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn"] Jul 10 13:51:13.776315 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:13.776262 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:28.006792 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.006690 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" podUID="172b992f-bc76-473f-a9e3-ae00bc1fcb66" containerName="main" containerID="cri-o://8ab9e90b25b8143d54504f03cc61e03ada4f98b3af03e22ba001ea11eb453d36" gracePeriod=45 Jul 10 13:51:28.139099 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.139071 2567 generic.go:356] "Generic (PLEG): container finished" podID="172b992f-bc76-473f-a9e3-ae00bc1fcb66" containerID="8ab9e90b25b8143d54504f03cc61e03ada4f98b3af03e22ba001ea11eb453d36" exitCode=0 Jul 10 13:51:28.139220 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.139152 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" event={"ID":"172b992f-bc76-473f-a9e3-ae00bc1fcb66","Type":"ContainerDied","Data":"8ab9e90b25b8143d54504f03cc61e03ada4f98b3af03e22ba001ea11eb453d36"} Jul 10 13:51:28.246564 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.246540 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:51:28.373198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373121 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-home\") pod \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " Jul 10 13:51:28.373198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373153 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-dshm\") pod \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " Jul 10 13:51:28.373198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373175 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-model-cache\") pod \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " Jul 10 13:51:28.373455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373236 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tls-certs\") pod \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " Jul 10 13:51:28.373455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373295 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tmp-dir\") pod \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " Jul 10 13:51:28.373455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373394 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-489ll\" (UniqueName: \"kubernetes.io/projected/172b992f-bc76-473f-a9e3-ae00bc1fcb66-kube-api-access-489ll\") pod \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\" (UID: \"172b992f-bc76-473f-a9e3-ae00bc1fcb66\") " Jul 10 13:51:28.373455 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373410 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-home" (OuterVolumeSpecName: "home") pod "172b992f-bc76-473f-a9e3-ae00bc1fcb66" (UID: "172b992f-bc76-473f-a9e3-ae00bc1fcb66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:28.373670 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373480 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-model-cache" (OuterVolumeSpecName: "model-cache") pod "172b992f-bc76-473f-a9e3-ae00bc1fcb66" (UID: "172b992f-bc76-473f-a9e3-ae00bc1fcb66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:28.373730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373686 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:28.373730 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373699 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:28.373820 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.373759 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "172b992f-bc76-473f-a9e3-ae00bc1fcb66" (UID: "172b992f-bc76-473f-a9e3-ae00bc1fcb66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:28.375287 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.375258 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-dshm" (OuterVolumeSpecName: "dshm") pod "172b992f-bc76-473f-a9e3-ae00bc1fcb66" (UID: "172b992f-bc76-473f-a9e3-ae00bc1fcb66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:28.375396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.375287 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172b992f-bc76-473f-a9e3-ae00bc1fcb66-kube-api-access-489ll" (OuterVolumeSpecName: "kube-api-access-489ll") pod "172b992f-bc76-473f-a9e3-ae00bc1fcb66" (UID: "172b992f-bc76-473f-a9e3-ae00bc1fcb66"). InnerVolumeSpecName "kube-api-access-489ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:28.375396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.375349 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "172b992f-bc76-473f-a9e3-ae00bc1fcb66" (UID: "172b992f-bc76-473f-a9e3-ae00bc1fcb66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:51:28.474614 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.474592 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:28.474614 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.474612 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:28.474743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.474622 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172b992f-bc76-473f-a9e3-ae00bc1fcb66-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:28.474743 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.474631 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-489ll\" (UniqueName: \"kubernetes.io/projected/172b992f-bc76-473f-a9e3-ae00bc1fcb66-kube-api-access-489ll\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:28.775665 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:28.775627 2567 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" podUID="e73b4b4e-1c42-4a05-a206-b11f6206786b" containerName="main" containerID="cri-o://e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63" gracePeriod=45 Jul 10 13:51:29.010854 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.010815 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:51:29.080190 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080127 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-tmp-dir\") pod \"e73b4b4e-1c42-4a05-a206-b11f6206786b\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " Jul 10 13:51:29.080190 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080171 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-dshm\") pod \"e73b4b4e-1c42-4a05-a206-b11f6206786b\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " Jul 10 13:51:29.080340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080191 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-model-cache\") pod \"e73b4b4e-1c42-4a05-a206-b11f6206786b\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " Jul 10 13:51:29.080340 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080282 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73b4b4e-1c42-4a05-a206-b11f6206786b-tls-certs\") pod \"e73b4b4e-1c42-4a05-a206-b11f6206786b\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " Jul 10 13:51:29.080429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080347 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-home\") pod \"e73b4b4e-1c42-4a05-a206-b11f6206786b\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " Jul 10 13:51:29.080429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080397 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e73b4b4e-1c42-4a05-a206-b11f6206786b" (UID: "e73b4b4e-1c42-4a05-a206-b11f6206786b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:29.080429 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080411 2567 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqt2l\" (UniqueName: \"kubernetes.io/projected/e73b4b4e-1c42-4a05-a206-b11f6206786b-kube-api-access-sqt2l\") pod \"e73b4b4e-1c42-4a05-a206-b11f6206786b\" (UID: \"e73b4b4e-1c42-4a05-a206-b11f6206786b\") " Jul 10 13:51:29.080538 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080423 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-model-cache" (OuterVolumeSpecName: "model-cache") pod "e73b4b4e-1c42-4a05-a206-b11f6206786b" (UID: "e73b4b4e-1c42-4a05-a206-b11f6206786b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:29.080627 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080605 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-home" (OuterVolumeSpecName: "home") pod "e73b4b4e-1c42-4a05-a206-b11f6206786b" (UID: "e73b4b4e-1c42-4a05-a206-b11f6206786b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:29.080741 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080726 2567 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-home\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.080799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080743 2567 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-tmp-dir\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.080799 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.080751 2567 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-model-cache\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.082222 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.082199 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-dshm" (OuterVolumeSpecName: "dshm") pod "e73b4b4e-1c42-4a05-a206-b11f6206786b" (UID: "e73b4b4e-1c42-4a05-a206-b11f6206786b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 13:51:29.082344 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.082325 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e73b4b4e-1c42-4a05-a206-b11f6206786b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e73b4b4e-1c42-4a05-a206-b11f6206786b" (UID: "e73b4b4e-1c42-4a05-a206-b11f6206786b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 13:51:29.082387 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.082350 2567 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73b4b4e-1c42-4a05-a206-b11f6206786b-kube-api-access-sqt2l" (OuterVolumeSpecName: "kube-api-access-sqt2l") pod "e73b4b4e-1c42-4a05-a206-b11f6206786b" (UID: "e73b4b4e-1c42-4a05-a206-b11f6206786b"). InnerVolumeSpecName "kube-api-access-sqt2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 13:51:29.145557 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.145537 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" Jul 10 13:51:29.145698 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.145580 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth" event={"ID":"172b992f-bc76-473f-a9e3-ae00bc1fcb66","Type":"ContainerDied","Data":"92a2cc5fb974ed72133fd0fe93cf6440022bfe56685682bfb2ad96a7c7a90eae"} Jul 10 13:51:29.145698 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.145629 2567 scope.go:117] "RemoveContainer" containerID="8ab9e90b25b8143d54504f03cc61e03ada4f98b3af03e22ba001ea11eb453d36" Jul 10 13:51:29.147036 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.147015 2567 generic.go:356] "Generic (PLEG): container finished" podID="e73b4b4e-1c42-4a05-a206-b11f6206786b" containerID="e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63" exitCode=0 Jul 10 13:51:29.147134 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.147073 2567 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" Jul 10 13:51:29.147198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.147071 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" event={"ID":"e73b4b4e-1c42-4a05-a206-b11f6206786b","Type":"ContainerDied","Data":"e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63"} Jul 10 13:51:29.147198 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.147184 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn" event={"ID":"e73b4b4e-1c42-4a05-a206-b11f6206786b","Type":"ContainerDied","Data":"1a4ac52e8fbaac8d9ec8d185dd29fa6e998329754dc9d8eec84bd692ec3cc784"} Jul 10 13:51:29.156545 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.156475 2567 scope.go:117] "RemoveContainer" containerID="e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63" Jul 10 13:51:29.165747 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.165730 2567 scope.go:117] "RemoveContainer" containerID="e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63" Jul 10 13:51:29.165999 ip-10-0-140-11 kubenswrapper[2567]: E0710 13:51:29.165982 2567 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63\": container with ID starting with e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63 not found: ID does not exist" containerID="e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63" Jul 10 13:51:29.166053 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.166005 2567 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63"} err="failed to get container status \"e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63\": rpc error: code = NotFound desc = could not find container \"e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63\": container with ID starting with e76d7dc1c01468a21ebfa4cd1e4f0a988fc341b66d40c20c31953b3a44f96b63 not found: ID does not exist" Jul 10 13:51:29.171425 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.171407 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth"] Jul 10 13:51:29.178232 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.178213 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-90a164/rate-test-model-90a164-kserve-7476498f75-4gdth"] Jul 10 13:51:29.181435 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.181418 2567 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e73b4b4e-1c42-4a05-a206-b11f6206786b-dshm\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.181504 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.181453 2567 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e73b4b4e-1c42-4a05-a206-b11f6206786b-tls-certs\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.181504 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.181463 2567 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqt2l\" (UniqueName: \"kubernetes.io/projected/e73b4b4e-1c42-4a05-a206-b11f6206786b-kube-api-access-sqt2l\") on node \"ip-10-0-140-11.ec2.internal\" DevicePath \"\"" Jul 10 13:51:29.188874 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.188856 2567 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn"] Jul 10 13:51:29.192013 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.191994 2567 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-efd2ad/rate-test-model-efd2ad-kserve-86f67f8f67-mt7pn"] Jul 10 13:51:29.951022 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.950989 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172b992f-bc76-473f-a9e3-ae00bc1fcb66" path="/var/lib/kubelet/pods/172b992f-bc76-473f-a9e3-ae00bc1fcb66/volumes" Jul 10 13:51:29.951386 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:29.951372 2567 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e73b4b4e-1c42-4a05-a206-b11f6206786b" path="/var/lib/kubelet/pods/e73b4b4e-1c42-4a05-a206-b11f6206786b/volumes" Jul 10 13:51:32.575003 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:32.574964 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:32.997128 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:32.997088 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:33.094655 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:33.094616 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:33.196765 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:33.196719 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:33.296642 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:33.296560 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.149582 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:39.149551 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.226338 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:39.226301 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.411528 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:39.411452 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.633383 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:39.633347 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:39.847492 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:39.847459 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:40.266226 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:40.266191 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:40.366996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:40.366962 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:40.473102 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:40.473069 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:40.625283 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:40.625197 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.389779 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:46.389738 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.490010 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:46.489972 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.656214 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:46.656127 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:46.888751 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:46.888690 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:51:52.766581 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:52.766550 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-2qd5x_b6816528-0fe2-474e-9ea8-8a201c8be438/manager/0.log" Jul 10 13:51:52.772649 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:52.772629 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-2qd5x_b6816528-0fe2-474e-9ea8-8a201c8be438/copy-manifests/0.log" Jul 10 13:51:53.427548 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:53.427503 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-p46pn_bb3a7c9c-c665-4802-bdf8-a4b1e11cdf3c/postgres/0.log" Jul 10 13:51:54.227179 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.227149 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4_b2576174-c9b6-4b39-baac-05982b9e5f4e/util/0.log" Jul 10 13:51:54.233858 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.233816 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4_b2576174-c9b6-4b39-baac-05982b9e5f4e/pull/0.log" Jul 10 13:51:54.240150 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.240127 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4_b2576174-c9b6-4b39-baac-05982b9e5f4e/extract/0.log" Jul 10 13:51:54.352293 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.352269 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc_02a4aba3-abee-4b36-bb31-5273afe623d0/util/0.log" Jul 10 13:51:54.358865 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.358829 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc_02a4aba3-abee-4b36-bb31-5273afe623d0/pull/0.log" Jul 10 13:51:54.365226 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.365205 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc_02a4aba3-abee-4b36-bb31-5273afe623d0/extract/0.log" Jul 10 13:51:54.475769 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.475749 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9_09b4ecc1-9b42-4c81-adc7-e378b8674627/util/0.log" Jul 10 13:51:54.482291 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.482236 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9_09b4ecc1-9b42-4c81-adc7-e378b8674627/pull/0.log" Jul 10 13:51:54.488375 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.488356 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9_09b4ecc1-9b42-4c81-adc7-e378b8674627/extract/0.log" Jul 10 13:51:54.599960 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.599940 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t_a977dd51-c671-4b36-891d-2ada964af0ea/util/0.log" Jul 10 13:51:54.606609 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.606588 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t_a977dd51-c671-4b36-891d-2ada964af0ea/pull/0.log" Jul 10 13:51:54.612956 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:54.612936 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t_a977dd51-c671-4b36-891d-2ada964af0ea/extract/0.log" Jul 10 13:51:55.933296 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:55.933266 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zf96b_a30b42e9-6fb7-406d-8933-31b41682c8e6/istio-proxy/0.log" Jul 10 13:51:56.062151 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:56.062101 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zmcd5_e82c6e60-f99b-4307-950c-46701a51a262/discovery/0.log" Jul 10 13:51:56.402101 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:56.402071 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/1.log" Jul 10 13:51:57.090448 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:57.090414 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6b6wh_2e69cbcf-7d18-4237-b9cc-c1fc369a6171/main/0.log" Jul 10 13:51:57.310391 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:51:57.310356 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5ct8m_4bc6ce88-6e33-4f1c-ae1b-35fa341da8a2/main/0.log" Jul 10 13:52:01.639348 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:01.639305 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:01.745649 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:01.745602 2567 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" limit=1024 Jul 10 13:52:04.465490 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:04.465459 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xhc8c_a311c9cc-4ec0-4099-a20c-e85c5e1c6801/global-pull-secret-syncer/0.log" Jul 10 13:52:04.571378 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:04.571344 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dc9tb_00558e71-8d16-44ce-aa14-d69cdf75fb72/konnectivity-agent/0.log" Jul 10 13:52:04.724170 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:04.724096 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-11.ec2.internal_87b229af0a3dd64d765568e457658e38/haproxy/0.log" Jul 10 13:52:09.384170 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.384142 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4_b2576174-c9b6-4b39-baac-05982b9e5f4e/extract/0.log" Jul 10 13:52:09.407434 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.407409 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4_b2576174-c9b6-4b39-baac-05982b9e5f4e/util/0.log" Jul 10 13:52:09.432254 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.432232 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759mq2q4_b2576174-c9b6-4b39-baac-05982b9e5f4e/pull/0.log" Jul 10 13:52:09.461499 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.461479 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc_02a4aba3-abee-4b36-bb31-5273afe623d0/extract/0.log" Jul 10 13:52:09.484550 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.484533 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc_02a4aba3-abee-4b36-bb31-5273afe623d0/util/0.log" Jul 10 13:52:09.507211 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.507191 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0q2jkc_02a4aba3-abee-4b36-bb31-5273afe623d0/pull/0.log" Jul 10 13:52:09.553358 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.553324 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9_09b4ecc1-9b42-4c81-adc7-e378b8674627/extract/0.log" Jul 10 13:52:09.574922 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.574901 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9_09b4ecc1-9b42-4c81-adc7-e378b8674627/util/0.log" Jul 10 13:52:09.595761 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.595744 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed735n2h9_09b4ecc1-9b42-4c81-adc7-e378b8674627/pull/0.log" Jul 10 13:52:09.630656 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.630633 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t_a977dd51-c671-4b36-891d-2ada964af0ea/extract/0.log" Jul 10 13:52:09.653170 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.653115 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t_a977dd51-c671-4b36-891d-2ada964af0ea/util/0.log" Jul 10 13:52:09.678343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:09.678321 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1sfc7t_a977dd51-c671-4b36-891d-2ada964af0ea/pull/0.log" Jul 10 13:52:12.342395 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.342362 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zd2f7_f9d576ed-1c8b-4be1-91ad-7e672b29f503/kube-state-metrics/0.log" Jul 10 13:52:12.393706 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.393675 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zd2f7_f9d576ed-1c8b-4be1-91ad-7e672b29f503/kube-rbac-proxy-main/0.log" Jul 10 13:52:12.465655 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.465632 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-zd2f7_f9d576ed-1c8b-4be1-91ad-7e672b29f503/kube-rbac-proxy-self/0.log" Jul 10 13:52:12.504047 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.504026 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5fb48b6c7b-pzfvp_5270714f-bf33-4505-9016-9ce0bd512888/metrics-server/0.log" Jul 10 13:52:12.529781 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.529758 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-5l6cs_e3743f48-b170-4b48-bdd8-72328f25ea78/monitoring-plugin/0.log" Jul 10 13:52:12.567396 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.567378 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n4shw_91110571-c3fa-49ac-a975-a00d7e2aee97/node-exporter/0.log" Jul 10 13:52:12.593321 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.593255 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n4shw_91110571-c3fa-49ac-a975-a00d7e2aee97/kube-rbac-proxy/0.log" Jul 10 13:52:12.617042 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:12.617024 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n4shw_91110571-c3fa-49ac-a975-a00d7e2aee97/init-textfile/0.log" Jul 10 13:52:13.156257 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.156218 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-x9kht_c9afcc22-2cac-4501-ac9f-c15fd97c03d6/prometheus-operator-admission-webhook/0.log" Jul 10 13:52:13.186168 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.186144 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-698d6d76c5-qzlpf_5a616786-9fdb-478c-a17c-12795174e260/telemeter-client/0.log" Jul 10 13:52:13.210921 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.210892 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-698d6d76c5-qzlpf_5a616786-9fdb-478c-a17c-12795174e260/reload/0.log" Jul 10 13:52:13.235244 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.235222 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-698d6d76c5-qzlpf_5a616786-9fdb-478c-a17c-12795174e260/kube-rbac-proxy/0.log" Jul 10 13:52:13.263585 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.263564 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c89c6669-58qj7_340b9c25-1e72-47c4-9815-a45f663607ae/thanos-query/0.log" Jul 10 13:52:13.290431 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.290413 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c89c6669-58qj7_340b9c25-1e72-47c4-9815-a45f663607ae/kube-rbac-proxy-web/0.log" Jul 10 13:52:13.314440 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.314420 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c89c6669-58qj7_340b9c25-1e72-47c4-9815-a45f663607ae/kube-rbac-proxy/0.log" Jul 10 13:52:13.336500 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.336481 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c89c6669-58qj7_340b9c25-1e72-47c4-9815-a45f663607ae/prom-label-proxy/0.log" Jul 10 13:52:13.359752 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.359734 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c89c6669-58qj7_340b9c25-1e72-47c4-9815-a45f663607ae/kube-rbac-proxy-rules/0.log" Jul 10 13:52:13.382773 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:13.382750 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c89c6669-58qj7_340b9c25-1e72-47c4-9815-a45f663607ae/kube-rbac-proxy-metrics/0.log" Jul 10 13:52:14.556035 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.555995 2567 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9"] Jul 10 13:52:14.556523 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.556468 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e73b4b4e-1c42-4a05-a206-b11f6206786b" containerName="main" Jul 10 13:52:14.556523 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.556487 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="e73b4b4e-1c42-4a05-a206-b11f6206786b" containerName="main" Jul 10 13:52:14.556523 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.556516 2567 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172b992f-bc76-473f-a9e3-ae00bc1fcb66" containerName="main" Jul 10 13:52:14.556523 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.556523 2567 state_mem.go:107] "Deleted CPUSet assignment" podUID="172b992f-bc76-473f-a9e3-ae00bc1fcb66" containerName="main" Jul 10 13:52:14.556675 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.556594 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="e73b4b4e-1c42-4a05-a206-b11f6206786b" containerName="main" Jul 10 13:52:14.556675 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.556605 2567 memory_manager.go:368] "RemoveStaleState removing state" podUID="172b992f-bc76-473f-a9e3-ae00bc1fcb66" containerName="main" Jul 10 13:52:14.559724 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.559707 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.562000 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.561979 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k76w9\"/\"kube-root-ca.crt\"" Jul 10 13:52:14.562124 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.562041 2567 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k76w9\"/\"default-dockercfg-wps8p\"" Jul 10 13:52:14.562808 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.562788 2567 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k76w9\"/\"openshift-service-ca.crt\"" Jul 10 13:52:14.568360 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.568322 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9"] Jul 10 13:52:14.648076 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.648052 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-sys\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.648185 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.648083 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pz85\" (UniqueName: \"kubernetes.io/projected/3105dd08-c04a-4de3-ac51-b5da96fc9b00-kube-api-access-4pz85\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.648185 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.648121 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-lib-modules\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.648259 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.648219 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-proc\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.648298 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.648257 2567 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-podres\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.748906 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.748869 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-sys\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749047 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.748913 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pz85\" (UniqueName: \"kubernetes.io/projected/3105dd08-c04a-4de3-ac51-b5da96fc9b00-kube-api-access-4pz85\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749047 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.748953 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-lib-modules\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749047 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.748990 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-proc\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749047 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.748990 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-sys\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749047 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.749027 2567 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-podres\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.749060 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-proc\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.749130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-podres\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.749217 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.749130 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3105dd08-c04a-4de3-ac51-b5da96fc9b00-lib-modules\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.756996 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.756971 2567 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pz85\" (UniqueName: \"kubernetes.io/projected/3105dd08-c04a-4de3-ac51-b5da96fc9b00-kube-api-access-4pz85\") pod \"perf-node-gather-daemonset-cw2n9\" (UID: \"3105dd08-c04a-4de3-ac51-b5da96fc9b00\") " pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:14.870273 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:14.870205 2567 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:15.001501 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:15.001472 2567 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9"] Jul 10 13:52:15.003728 ip-10-0-140-11 kubenswrapper[2567]: W0710 13:52:15.003698 2567 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3105dd08_c04a_4de3_ac51_b5da96fc9b00.slice/crio-ad6a77125f66f8135625b8e824a7024af5003efee046acc7e01ea5a4b7a2e70a WatchSource:0}: Error finding container ad6a77125f66f8135625b8e824a7024af5003efee046acc7e01ea5a4b7a2e70a: Status 404 returned error can't find the container with id ad6a77125f66f8135625b8e824a7024af5003efee046acc7e01ea5a4b7a2e70a Jul 10 13:52:15.375175 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:15.375136 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" event={"ID":"3105dd08-c04a-4de3-ac51-b5da96fc9b00","Type":"ContainerStarted","Data":"daf1e5fc40e45ff59751b5a1d28a7baf4c244ddc398794612721d54277a95d98"} Jul 10 13:52:15.375371 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:15.375196 2567 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" event={"ID":"3105dd08-c04a-4de3-ac51-b5da96fc9b00","Type":"ContainerStarted","Data":"ad6a77125f66f8135625b8e824a7024af5003efee046acc7e01ea5a4b7a2e70a"} Jul 10 13:52:15.375371 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:15.375333 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:15.393803 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:15.393762 2567 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" podStartSLOduration=1.39374801 podStartE2EDuration="1.39374801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 13:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 13:52:15.391931358 +0000 UTC m=+2516.108308988" watchObservedRunningTime="2026-07-10 13:52:15.39374801 +0000 UTC m=+2516.110125636" Jul 10 13:52:17.479454 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:17.479415 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdk2c_a787e51b-809a-4735-9f40-2d3331f1824d/dns/0.log" Jul 10 13:52:17.503311 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:17.503284 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdk2c_a787e51b-809a-4735-9f40-2d3331f1824d/kube-rbac-proxy/0.log" Jul 10 13:52:17.556492 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:17.556471 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-r4gbb_615c6ac8-7da2-4ea6-9c80-4a8e5f7d0fde/dns-node-resolver/0.log" Jul 10 13:52:18.140646 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:18.140619 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6fd5697954-ngqvd_8aed2557-b2e1-499b-807e-4d61ad1f81ef/registry/0.log" Jul 10 13:52:18.188414 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:18.188386 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-65xfz_f6a6b054-52ca-4dfe-abf5-2ef50b8a3226/node-ca/0.log" Jul 10 13:52:19.217430 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:19.217402 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zf96b_a30b42e9-6fb7-406d-8933-31b41682c8e6/istio-proxy/0.log" Jul 10 13:52:19.426324 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:19.426298 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zmcd5_e82c6e60-f99b-4307-950c-46701a51a262/discovery/0.log" Jul 10 13:52:19.512767 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:19.512695 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/0.log" Jul 10 13:52:19.571370 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:19.571346 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/1.log" Jul 10 13:52:20.510889 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:20.510864 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hnqr4_50f2aa9c-9afd-42a4-9155-2d2006ba5b50/serve-healthcheck-canary/0.log" Jul 10 13:52:21.158997 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:21.158971 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw8td_29e1b4e5-a9ec-44f7-b0a4-e40181929e18/kube-rbac-proxy/0.log" Jul 10 13:52:21.183522 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:21.183503 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw8td_29e1b4e5-a9ec-44f7-b0a4-e40181929e18/exporter/0.log" Jul 10 13:52:21.208248 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:21.208228 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kw8td_29e1b4e5-a9ec-44f7-b0a4-e40181929e18/extractor/0.log" Jul 10 13:52:21.389556 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:21.389533 2567 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k76w9/perf-node-gather-daemonset-cw2n9" Jul 10 13:52:24.129172 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:24.129135 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-2qd5x_b6816528-0fe2-474e-9ea8-8a201c8be438/manager/0.log" Jul 10 13:52:24.163568 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:24.163536 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9ff96f8f8-2qd5x_b6816528-0fe2-474e-9ea8-8a201c8be438/copy-manifests/0.log" Jul 10 13:52:25.134059 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:25.134027 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zf96b_a30b42e9-6fb7-406d-8933-31b41682c8e6/istio-proxy/0.log" Jul 10 13:52:25.310930 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:25.310900 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zmcd5_e82c6e60-f99b-4307-950c-46701a51a262/discovery/0.log" Jul 10 13:52:25.362227 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:25.362202 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/0.log" Jul 10 13:52:25.421233 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:25.421205 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/1.log" Jul 10 13:52:25.559489 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:25.559456 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cffcb8bcf-vt6mx_4781f44d-5278-46fc-8ec8-6873630c39ef/manager/0.log" Jul 10 13:52:25.585500 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:25.585475 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-wzhvx_118abeaa-67b0-432c-a763-1f8a2efd5b5d/openshift-lws-operator/0.log" Jul 10 13:52:26.683801 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:26.683767 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4zf96b_a30b42e9-6fb7-406d-8933-31b41682c8e6/istio-proxy/0.log" Jul 10 13:52:26.868014 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:26.867987 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-zmcd5_e82c6e60-f99b-4307-950c-46701a51a262/discovery/0.log" Jul 10 13:52:26.923512 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:26.923483 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/0.log" Jul 10 13:52:26.984610 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:26.984580 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-g7mql_236b4ab9-9245-43b0-bf7d-b58db9ac5895/istio-proxy/1.log" Jul 10 13:52:32.108367 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.108337 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/kube-multus-additional-cni-plugins/0.log" Jul 10 13:52:32.131553 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.131513 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/egress-router-binary-copy/0.log" Jul 10 13:52:32.154610 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.154587 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/cni-plugins/0.log" Jul 10 13:52:32.175058 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.175040 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/bond-cni-plugin/0.log" Jul 10 13:52:32.199343 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.199321 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/routeoverride-cni/0.log" Jul 10 13:52:32.225368 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.225350 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/whereabouts-cni-bincopy/0.log" Jul 10 13:52:32.250933 ip-10-0-140-11 kubenswrapper[2567]: I0710 13:52:32.250917 2567 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-98dtr_d9700277-f8d3-4ca9-8da5-7f728bfbd9a5/whereabouts-cni/0.log"