Sep 10 14:19:42.032072 ip-10-0-135-236 systemd[1]: Starting Kubernetes Kubelet... Sep 10 14:19:42.421672 ip-10-0-135-236 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:19:42.421672 ip-10-0-135-236 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 14:19:42.421672 ip-10-0-135-236 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:19:42.421672 ip-10-0-135-236 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 14:19:42.421672 ip-10-0-135-236 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 14:19:42.422547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.422466 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 14:19:42.426095 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426080 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:19:42.426095 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426094 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426097 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426101 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426104 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426106 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426109 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426113 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426116 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426118 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426121 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426124 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426138 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426141 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426144 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426146 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426149 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426152 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426154 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426157 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426159 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:19:42.426164 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426162 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426165 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426167 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426170 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426173 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426176 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426179 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426181 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426183 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426188 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426191 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426194 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426198 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426200 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426203 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426206 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426209 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426211 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426214 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:19:42.426639 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426216 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426219 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426221 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426224 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426226 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426234 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426237 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426239 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426242 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426245 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426249 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426251 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426254 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426257 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426259 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426265 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426269 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426272 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426275 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426278 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:19:42.427120 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426281 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426283 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426286 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426288 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426291 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426293 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426296 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426299 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426301 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426304 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426306 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426309 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426311 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426314 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426316 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426319 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426322 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426325 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426333 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:19:42.427589 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426336 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426338 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426341 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426343 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426347 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426349 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426352 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426354 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426357 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426360 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426363 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426365 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426368 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426371 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426373 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426376 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426378 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426381 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426384 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.426387 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:19:42.428103 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427932 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427940 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427944 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427947 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427950 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427953 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427955 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427958 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427961 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427964 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427966 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427969 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427972 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427974 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427977 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427979 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427982 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427984 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:19:42.428571 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427987 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427991 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427995 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.427998 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428001 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428004 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428007 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428010 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428013 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428016 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428018 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428020 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428023 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428025 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428028 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428032 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428035 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428038 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428042 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:19:42.429019 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428045 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428048 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428050 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428053 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428055 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428058 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428061 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428063 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428066 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428069 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428071 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428074 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428077 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428079 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428082 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428084 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428087 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428090 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428092 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428097 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:19:42.429475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428100 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428103 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428106 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428109 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428112 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428115 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428117 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428120 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428123 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428126 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428128 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428131 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428133 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428136 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428138 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428141 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428143 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428146 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428148 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428150 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:19:42.430008 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428153 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428156 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428160 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428162 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428165 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428168 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428170 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428173 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428175 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428178 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428181 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428183 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428185 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428188 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428190 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428193 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428195 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428198 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428200 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428202 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:19:42.430517 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428205 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428207 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428529 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428534 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428536 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428539 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428542 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428545 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428547 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428549 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428552 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428554 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428557 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428560 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428564 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428567 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428571 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428574 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428578 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428580 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:19:42.431034 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428583 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428586 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428589 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428591 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428594 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428597 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428601 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428604 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428607 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428610 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428613 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428616 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428619 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428622 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428624 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428627 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428629 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428632 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428634 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:19:42.431519 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428637 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428639 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428641 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428644 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428647 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428649 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428651 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428655 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428658 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428660 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428663 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428665 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428668 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428671 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428674 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428677 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428679 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428681 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428684 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428686 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:19:42.432013 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428689 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428691 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428694 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428696 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428699 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428701 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428704 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428707 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428709 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428712 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428714 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428717 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428719 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428721 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428724 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428726 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428729 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428731 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428733 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428737 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:19:42.432480 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428741 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428743 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428746 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428749 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428751 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428754 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428756 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428759 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428761 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428764 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428766 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428769 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428771 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428773 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428776 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428778 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428781 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428783 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428801 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428804 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:19:42.432996 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428807 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.428810 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428883 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428926 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428933 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428938 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428941 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428945 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428949 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428953 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428955 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428960 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428964 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428970 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428973 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428977 2576 flags.go:64] FLAG: --cgroup-root="" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428979 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428982 2576 flags.go:64] FLAG: --client-ca-file="" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428985 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428988 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428991 2576 flags.go:64] FLAG: --cluster-domain="" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428994 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.428997 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429000 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 14:19:42.433471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429003 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429007 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429010 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429013 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429016 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429019 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429022 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429025 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429029 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429032 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429035 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429039 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429042 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429044 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429047 2576 flags.go:64] FLAG: --enable-server="true" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429050 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429055 2576 flags.go:64] FLAG: --event-burst="100" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429058 2576 flags.go:64] FLAG: --event-qps="50" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429061 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429064 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429067 2576 flags.go:64] FLAG: --eviction-hard="" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429071 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429074 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429077 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429080 2576 flags.go:64] FLAG: --eviction-soft="" Sep 10 14:19:42.434095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429082 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429085 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429088 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429090 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429093 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429096 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429098 2576 flags.go:64] FLAG: --feature-gates="" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429102 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429105 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429107 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429110 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429113 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429116 2576 flags.go:64] FLAG: --help="false" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429119 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429122 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429125 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429128 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429131 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429134 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429137 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429139 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429142 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429145 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 14:19:42.434685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429147 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429150 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429153 2576 flags.go:64] FLAG: --kube-reserved="" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429156 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429158 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429161 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429164 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429167 2576 flags.go:64] FLAG: --lock-file="" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429170 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429173 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429176 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429181 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429183 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429186 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429189 2576 flags.go:64] FLAG: --logging-format="text" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429191 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429194 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429197 2576 flags.go:64] FLAG: --manifest-url="" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429200 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429204 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429207 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429211 2576 flags.go:64] FLAG: --max-pods="110" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429214 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429219 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429222 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 14:19:42.435268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429225 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429228 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429231 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429234 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429241 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429243 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429246 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429249 2576 flags.go:64] FLAG: --pod-cidr="" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429252 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429257 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429259 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429262 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429265 2576 flags.go:64] FLAG: --port="10250" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429268 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429270 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01d23b89aa7259656" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429274 2576 flags.go:64] FLAG: --qos-reserved="" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429277 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429279 2576 flags.go:64] FLAG: --register-node="true" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429282 2576 flags.go:64] FLAG: --register-with-taints="" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429285 2576 flags.go:64] FLAG: --registry-burst="10" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429288 2576 flags.go:64] FLAG: --registry-qps="5" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429291 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429293 2576 flags.go:64] FLAG: --reserved-memory="" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429300 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429303 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 14:19:42.435964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429305 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429308 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429311 2576 flags.go:64] FLAG: --runonce="false" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429313 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429316 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429320 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429323 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429325 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429328 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429331 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429334 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429336 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429339 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429342 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429344 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429347 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429350 2576 flags.go:64] FLAG: --system-cgroups="" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429352 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429357 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429360 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429363 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429366 2576 flags.go:64] FLAG: --tls-min-version="" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429369 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429372 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429375 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 14:19:42.436602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429378 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429381 2576 flags.go:64] FLAG: --v="2" Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429385 2576 flags.go:64] FLAG: --version="false" Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429388 2576 flags.go:64] FLAG: --vmodule="" Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429392 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.429395 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429501 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429505 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429508 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429510 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429513 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429516 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429520 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429522 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429525 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429527 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429529 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429532 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429534 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429537 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429539 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:19:42.437492 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429542 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429544 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429546 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429549 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429553 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429555 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429558 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429560 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429563 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429565 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429568 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429571 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429573 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429576 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429578 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429581 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429583 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429586 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429588 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429591 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:19:42.438222 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429593 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429596 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429598 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429602 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429604 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429607 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429610 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429619 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429622 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429625 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429627 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429630 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429632 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429635 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429638 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429640 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429644 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429647 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429668 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429671 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:19:42.438717 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429674 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429676 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429679 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429682 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429685 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429688 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429690 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429693 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429695 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429698 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429700 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429703 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429706 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429708 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429711 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429716 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429719 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429723 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429726 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:19:42.439224 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429729 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429731 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429734 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429738 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429741 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429744 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429746 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429749 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429752 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429755 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429758 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429760 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429763 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429766 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429768 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429771 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429774 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429777 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429780 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429784 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:19:42.439668 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429798 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429802 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429805 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429807 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.429810 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.430487 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.438710 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.438724 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.438737 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.438743 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438816 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438822 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438825 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438828 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438831 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:19:42.440154 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438847 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438850 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438852 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438855 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438858 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438860 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438863 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438865 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438868 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438870 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438873 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438875 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438878 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438881 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438883 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438891 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438893 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438896 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438899 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438901 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:19:42.440547 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438904 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438907 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438910 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438913 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438916 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438919 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438923 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438927 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438930 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438933 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438936 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438938 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438941 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438943 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438947 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438951 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438954 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438956 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438959 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:19:42.441088 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438961 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438964 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438967 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438970 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438972 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438975 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438977 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438980 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438982 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438985 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438987 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438990 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438992 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438995 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.438998 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439000 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439003 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439006 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439009 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439011 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:19:42.441538 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439014 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439016 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439019 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439021 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439024 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439027 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439029 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439032 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439034 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439037 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439039 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439041 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439044 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439046 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439049 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439052 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439055 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439057 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439059 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439062 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:19:42.442057 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439065 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439067 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439069 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439072 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439075 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439077 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439080 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439083 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439086 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439089 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439092 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439094 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439097 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439099 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439103 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:19:42.442531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.439108 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439230 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439236 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439239 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439241 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439244 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439247 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439250 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439252 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439255 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439257 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439260 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439262 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439266 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439268 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439270 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439273 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439275 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439278 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439280 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 14:19:42.442924 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439283 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439285 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439289 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439293 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439295 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439298 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439301 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439304 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439307 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439309 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439312 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439315 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439319 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439322 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439325 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439327 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439330 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439332 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439335 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 14:19:42.443405 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439337 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439339 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439342 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439344 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439347 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439349 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439352 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439355 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439358 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439360 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439363 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439365 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439368 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439370 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439373 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439375 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439378 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439380 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439383 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439385 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 14:19:42.443914 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439388 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439390 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439392 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439395 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439397 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439400 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439403 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439406 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439409 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439412 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439415 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439418 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439421 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439423 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439426 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439428 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439430 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439433 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439435 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 14:19:42.444398 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439438 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439440 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439443 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439446 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439448 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439450 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439453 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439455 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439458 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439460 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439462 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439465 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439468 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439470 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439472 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439475 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439477 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439480 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439482 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439484 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 14:19:42.444855 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439487 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:42.439489 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.439493 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.439596 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.442340 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.443401 2576 server.go:1019] "Starting client certificate rotation" Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.443497 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 14:19:42.445328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.443526 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 14:19:42.446896 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.446884 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 14:19:42.452138 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.452123 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 14:19:42.462447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.462425 2576 log.go:25] "Validated CRI v1 runtime API" Sep 10 14:19:42.466231 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.466217 2576 log.go:25] "Validated CRI v1 image API" Sep 10 14:19:42.467126 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.467110 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 14:19:42.471024 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.471006 2576 fs.go:135] Filesystem UUIDs: map[0cd3af56-ea74-4159-8b2a-17f6fbc6d165:/dev/nvme0n1p3 6b531367-d1e8-497c-a11d-13c376518b52:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Sep 10 14:19:42.471086 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.471023 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 14:19:42.473864 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.473840 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 14:19:42.476765 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.476662 2576 manager.go:216] Machine: {Timestamp:2026-09-10 14:19:42.475367258 +0000 UTC m=+0.329275924 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3173180 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29ddb394bbe83b66223143f795c524 SystemUUID:ec29ddb3-94bb-e83b-6622-3143f795c524 BootID:4af40fd8-f29c-4982-a383-02ab08c9266e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:58:df:1b:9b:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:58:df:1b:9b:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:90:f3:0f:74:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 14:19:42.476765 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.476762 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 14:19:42.476889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.476845 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 14:19:42.478394 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.478373 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 14:19:42.478526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.478397 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 14:19:42.479715 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.479705 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 14:19:42.479755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.479717 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 14:19:42.479755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.479724 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 14:19:42.480561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.480551 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 14:19:42.481908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.481899 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 14:19:42.484739 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484729 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 14:19:42.484876 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484866 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 14:19:42.484913 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484895 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 10 14:19:42.484913 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484905 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 14:19:42.484969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484922 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 14:19:42.484969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484931 2576 kubelet.go:398] "Adding apiserver pod source" Sep 10 14:19:42.484969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.484939 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 14:19:42.486249 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.486230 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 14:19:42.486335 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.486254 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 14:19:42.489029 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.489012 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 14:19:42.491316 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491303 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 14:19:42.491366 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491333 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 14:19:42.491499 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491488 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 14:19:42.491533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491508 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 14:19:42.491533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491515 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 14:19:42.491533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491521 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 14:19:42.491533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491528 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 14:19:42.491533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.491533 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 14:19:42.492094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492086 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 14:19:42.492094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492094 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 14:19:42.492146 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492100 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 14:19:42.492146 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492106 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 14:19:42.492146 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492113 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 14:19:42.492146 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492121 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 14:19:42.492705 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492698 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 14:19:42.492705 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.492706 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 14:19:42.495445 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.495421 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 14:19:42.495583 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.495561 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 14:19:42.495673 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.495658 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 14:19:42.496579 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.496563 2576 server.go:1268] "Started kubelet" Sep 10 14:19:42.496722 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.496647 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 14:19:42.496876 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.496703 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 14:19:42.496876 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.496750 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 14:19:42.497488 ip-10-0-135-236 systemd[1]: Started Kubernetes Kubelet. Sep 10 14:19:42.498564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.497844 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 14:19:42.499021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.499003 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9xv8" Sep 10 14:19:42.499329 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.499315 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 10 14:19:42.501331 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.501301 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 14:19:42.502080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.502063 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 14:19:42.502168 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.502077 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 14:19:42.502588 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.502539 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:42.502865 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.502850 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 14:19:42.502964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.502852 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 14:19:42.502964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.502965 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 14:19:42.503119 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503044 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 10 14:19:42.503119 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503053 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 10 14:19:42.503576 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503561 2576 factory.go:153] Registering CRI-O factory Sep 10 14:19:42.503576 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503578 2576 factory.go:223] Registration of the crio container factory successfully Sep 10 14:19:42.503719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503621 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 14:19:42.503719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503630 2576 factory.go:55] Registering systemd factory Sep 10 14:19:42.503719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503638 2576 factory.go:223] Registration of the systemd container factory successfully Sep 10 14:19:42.503719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503652 2576 factory.go:103] Registering Raw factory Sep 10 14:19:42.503719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.503664 2576 manager.go:1196] Started watching for new ooms in manager Sep 10 14:19:42.504054 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.504042 2576 manager.go:349] Starting recovery of all containers Sep 10 14:19:42.504853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.504833 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h9xv8" Sep 10 14:19:42.507779 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.507751 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 14:19:42.509391 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.508409 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-236.ec2.internal.18d3fb7af41ffd73 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-236.ec2.internal,UID:ip-10-0-135-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-236.ec2.internal,},FirstTimestamp:2026-09-10 14:19:42.496537971 +0000 UTC m=+0.350446637,LastTimestamp:2026-09-10 14:19:42.496537971 +0000 UTC m=+0.350446637,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-236.ec2.internal,}" Sep 10 14:19:42.509541 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.509504 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 14:19:42.516356 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.515565 2576 manager.go:354] Recovery completed Sep 10 14:19:42.519999 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.519895 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.522053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522037 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.522120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522063 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.522120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522073 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.522682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522667 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 14:19:42.522682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522681 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 14:19:42.522823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522695 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.522823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.522698 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 10 14:19:42.523492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.523404 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.523492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.523425 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.523492 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.523446 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.524208 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.524196 2576 policy_none.go:49] "None policy: Start" Sep 10 14:19:42.524246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.524213 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 14:19:42.524246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.524226 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 14:19:42.525382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.525370 2576 policy_none.go:47] "Start" Sep 10 14:19:42.547014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547002 2576 manager.go:337] "Starting Device Plugin manager" Sep 10 14:19:42.547073 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.547033 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 14:19:42.547073 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547042 2576 server.go:87] "Starting device plugin registration server" Sep 10 14:19:42.547328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547315 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 14:19:42.547391 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547327 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 14:19:42.547480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547452 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 14:19:42.547560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547535 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 14:19:42.547560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.547549 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 14:19:42.547909 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.547893 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 14:19:42.547959 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.547941 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:42.593906 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.593891 2576 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2609 into the system container \"/system.slice\": " Sep 10 14:19:42.648288 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.648268 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.649172 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.649157 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.649234 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.649183 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.649234 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.649193 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.649234 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.649218 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.657484 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.657470 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.657570 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.657492 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-236.ec2.internal\": node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:42.674462 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.674400 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:42.752654 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.752630 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 14:19:42.753960 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.753945 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 14:19:42.754030 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.753992 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 14:19:42.754030 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.754015 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 14:19:42.754138 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.754102 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 14:19:42.756694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.756678 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:19:42.775118 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.775094 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:42.854507 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.854482 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal"] Sep 10 14:19:42.854576 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.854553 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.856052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.856036 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.856118 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.856064 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.856118 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.856073 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.857294 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.857283 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.857427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.857412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.857475 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.857441 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.858079 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.858064 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.858158 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.858094 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.858158 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.858108 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.858583 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.858568 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.858672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.858592 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.858672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.858602 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.859866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.859852 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.859929 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.859878 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 14:19:42.861307 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.861276 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientMemory" Sep 10 14:19:42.861307 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.861299 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 14:19:42.861435 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.861311 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeHasSufficientPID" Sep 10 14:19:42.875988 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.875970 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:42.888280 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.888263 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-236.ec2.internal\" not found" node="ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.892615 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.892599 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-236.ec2.internal\" not found" node="ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.905067 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.905049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a7a76a58af3f5a735a46f27cbfd95e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal\" (UID: \"7a7a76a58af3f5a735a46f27cbfd95e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.905121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.905071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72e1c97d5f33b9f41328e5f5e35a0ce0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-236.ec2.internal\" (UID: \"72e1c97d5f33b9f41328e5f5e35a0ce0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.905158 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:42.905125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a7a76a58af3f5a735a46f27cbfd95e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal\" (UID: \"7a7a76a58af3f5a735a46f27cbfd95e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:42.976666 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:42.976626 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:43.006180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.006159 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a7a76a58af3f5a735a46f27cbfd95e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal\" (UID: \"7a7a76a58af3f5a735a46f27cbfd95e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.006228 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.006187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a7a76a58af3f5a735a46f27cbfd95e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal\" (UID: \"7a7a76a58af3f5a735a46f27cbfd95e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.006228 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.006204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72e1c97d5f33b9f41328e5f5e35a0ce0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-236.ec2.internal\" (UID: \"72e1c97d5f33b9f41328e5f5e35a0ce0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.006305 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.006249 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7a7a76a58af3f5a735a46f27cbfd95e9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal\" (UID: \"7a7a76a58af3f5a735a46f27cbfd95e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.006305 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.006254 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72e1c97d5f33b9f41328e5f5e35a0ce0-config\") pod \"kube-apiserver-proxy-ip-10-0-135-236.ec2.internal\" (UID: \"72e1c97d5f33b9f41328e5f5e35a0ce0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.006305 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.006274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a7a76a58af3f5a735a46f27cbfd95e9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal\" (UID: \"7a7a76a58af3f5a735a46f27cbfd95e9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.077322 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.077297 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:43.177865 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.177829 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:43.190992 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.190974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.195616 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.195596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.280364 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.280290 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:43.381344 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.381311 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-236.ec2.internal\" not found" Sep 10 14:19:43.434053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.434031 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:19:43.442661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.442637 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 14:19:43.442766 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.442749 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 14:19:43.442829 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.442806 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 14:19:43.486056 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.486039 2576 apiserver.go:52] "Watching apiserver" Sep 10 14:19:43.502533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.502504 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 14:19:43.502618 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.502532 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.507746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.507706 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 14:14:43 +0000 UTC" deadline="2028-06-22 13:38:37.122605854 +0000 UTC" Sep 10 14:19:43.507746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.507743 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15623h18m53.614864954s" Sep 10 14:19:43.509485 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.509464 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 14:19:43.509804 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.509784 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-7h85z","kube-system/global-pull-secret-syncer-9pl2k","openshift-multus/multus-hmvj6","openshift-multus/network-metrics-daemon-q996v","openshift-network-operator/iptables-alerter-kfgv6","openshift-ovn-kubernetes/ovnkube-node-jrjvz","kube-system/konnectivity-agent-2m7lm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7","openshift-cluster-node-tuning-operator/tuned-q4qfg","openshift-dns/node-resolver-8998p","openshift-image-registry/node-ca-pwhpf","openshift-multus/multus-additional-cni-plugins-pv52q"] Sep 10 14:19:43.511378 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.511360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:43.511465 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.511446 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:43.512597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.512580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.512684 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.512645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:43.514107 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.514093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.515250 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.515235 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:43.515323 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.515301 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:43.516480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.516463 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.516746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.516725 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kf4dz\"" Sep 10 14:19:43.516823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.516767 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.516823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.516775 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 14:19:43.516937 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.516879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.516937 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.516908 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 14:19:43.518068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.518054 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.519279 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.519265 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.519461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.519443 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.519534 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.519513 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-65vj5\"" Sep 10 14:19:43.519653 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.519638 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 14:19:43.519770 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.519757 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.521339 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521316 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.521507 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521447 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f88xw\"" Sep 10 14:19:43.521813 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521785 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 14:19:43.521910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521850 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.521910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521869 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 14:19:43.521910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521887 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" Sep 10 14:19:43.521910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521871 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 14:19:43.522066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.521939 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 14:19:43.522233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.522212 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 14:19:43.523099 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.523076 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.523928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.523291 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 14:19:43.523928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.523518 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 14:19:43.523928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.523844 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7np9z\"" Sep 10 14:19:43.524133 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.524019 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.525259 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.524371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q4hsx\"" Sep 10 14:19:43.525259 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.524383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 14:19:43.525259 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.524595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.525461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.525443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.527884 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.527867 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.528655 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.528638 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.528753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.528664 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.528753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.528668 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-m5tqk\"" Sep 10 14:19:43.529322 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.529303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.530527 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.530477 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hmrq4\"" Sep 10 14:19:43.530527 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.530494 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.530734 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.530718 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.530808 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.530789 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal"] Sep 10 14:19:43.530926 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.530913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.531876 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.531854 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 14:19:43.531969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.531854 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 14:19:43.531969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.531946 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7dq9n\"" Sep 10 14:19:43.532085 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.531989 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 14:19:43.532085 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.532054 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 14:19:43.532183 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.532151 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal"] Sep 10 14:19:43.533298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.533285 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 14:19:43.533685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.533668 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 14:19:43.533778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.533689 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f6gwc\"" Sep 10 14:19:43.568254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.568230 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 14:19:43.568343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.568259 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:19:43.594960 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.594944 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cdz99" Sep 10 14:19:43.603766 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.603751 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 14:19:43.605973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.605953 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cdz99" Sep 10 14:19:43.609302 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609277 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-device-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.609414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609325 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1aa96c3-14fa-4594-ba74-97b26ff67379-host\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.609414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-etc-kubernetes\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.609414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcf8z\" (UniqueName: \"kubernetes.io/projected/16991ff6-9847-4586-b642-2db22ae33842-kube-api-access-rcf8z\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:43.609592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-slash\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.609592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-var-lib-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.609592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26d0251c-9cab-48df-bd56-a6724f77f293-hosts-file\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.609592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d0251c-9cab-48df-bd56-a6724f77f293-tmp-dir\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.609809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72521405-0340-47a7-85e1-770cbe06b42e-iptables-alerter-script\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.609809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdf96\" (UniqueName: \"kubernetes.io/projected/72521405-0340-47a7-85e1-770cbe06b42e-kube-api-access-sdf96\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.609809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-systemd\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.609809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysctl-d\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.609809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-sys\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-cnibin\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-cni-bin\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysctl-conf\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.609959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-cni-bin\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.610074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e1410e4-456b-4092-afc8-e850a944a16a-ovn-node-metrics-cert\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610181 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-ovnkube-script-lib\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610209 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fa850c7-5397-4be3-80a9-2afab7eca821-kubelet-config\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610240 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/563c20c6-a7e1-412b-8682-95009c41620c-multus-daemon-config\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.610482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610293 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-kubelet\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.610788 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.610598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-env-overrides\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.612586 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.611662 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-registration-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.612714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.612609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysconfig\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.612714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.612649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8vl\" (UniqueName: \"kubernetes.io/projected/a1aa96c3-14fa-4594-ba74-97b26ff67379-kube-api-access-vp8vl\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.612714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.612710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-cni-netd\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.612881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.612814 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-lib-modules\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.612943 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.612926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-cnibin\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.612998 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.612965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-netns\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613013 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-conf-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613104 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-log-socket\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.613461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.613532 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613480 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-socket-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.613532 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-modprobe-d\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.613532 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-var-lib-kubelet\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-socket-dir-parent\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-kubelet\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8h8x\" (UniqueName: \"kubernetes.io/projected/563c20c6-a7e1-412b-8682-95009c41620c-kube-api-access-x8h8x\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613618 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1aa96c3-14fa-4594-ba74-97b26ff67379-serviceca\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-k8s-cni-cncf-io\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-cni-multus\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.613682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-multus-certs\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72521405-0340-47a7-85e1-770cbe06b42e-host-slash\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-systemd-units\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/87136eb6-b040-4611-9150-8308af770a4b-konnectivity-ca\") pod \"konnectivity-agent-2m7lm\" (UID: \"87136eb6-b040-4611-9150-8308af770a4b\") " pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-run-netns\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613858 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n52n4\" (UniqueName: \"kubernetes.io/projected/7e1410e4-456b-4092-afc8-e850a944a16a-kube-api-access-n52n4\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613913 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gsmb\" (UniqueName: \"kubernetes.io/projected/26d0251c-9cab-48df-bd56-a6724f77f293-kube-api-access-2gsmb\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.613994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-hostroot\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-etc-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.614084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-sys-fs\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlf5t\" (UniqueName: \"kubernetes.io/projected/1b325953-2f74-4fda-bc5b-cda5dc5d7290-kube-api-access-rlf5t\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614147 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-kubernetes\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-systemd\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614195 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/907eeeb1-e9b3-49df-a466-7eadf1141ae4-tmp\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614215 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-system-cni-dir\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-ovn\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-node-log\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-run\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-host\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjdtr\" (UniqueName: \"kubernetes.io/projected/907eeeb1-e9b3-49df-a466-7eadf1141ae4-kube-api-access-mjdtr\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614381 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fa850c7-5397-4be3-80a9-2afab7eca821-dbus\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnrkb\" (UniqueName: \"kubernetes.io/projected/048e8329-979a-4219-8b27-c2f7a41ed82d-kube-api-access-vnrkb\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-system-cni-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/563c20c6-a7e1-412b-8682-95009c41620c-cni-binary-copy\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.614531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-tuned\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.615121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/87136eb6-b040-4611-9150-8308af770a4b-agent-certs\") pod \"konnectivity-agent-2m7lm\" (UID: \"87136eb6-b040-4611-9150-8308af770a4b\") " pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.615121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-cni-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.615121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.615121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.614608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-ovnkube-config\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.715803 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715760 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fa850c7-5397-4be3-80a9-2afab7eca821-dbus\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.715803 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnrkb\" (UniqueName: \"kubernetes.io/projected/048e8329-979a-4219-8b27-c2f7a41ed82d-kube-api-access-vnrkb\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-system-cni-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/563c20c6-a7e1-412b-8682-95009c41620c-cni-binary-copy\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-tuned\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715880 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7fa850c7-5397-4be3-80a9-2afab7eca821-dbus\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/87136eb6-b040-4611-9150-8308af770a4b-agent-certs\") pod \"konnectivity-agent-2m7lm\" (UID: \"87136eb6-b040-4611-9150-8308af770a4b\") " pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.715935 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-cni-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.715994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-system-cni-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.716044 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:44.216001554 +0000 UTC m=+2.069910214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-cni-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-ovnkube-config\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716113 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-device-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1aa96c3-14fa-4594-ba74-97b26ff67379-host\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-etc-kubernetes\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcf8z\" (UniqueName: \"kubernetes.io/projected/16991ff6-9847-4586-b642-2db22ae33842-kube-api-access-rcf8z\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-slash\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716230 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a1aa96c3-14fa-4594-ba74-97b26ff67379-host\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716237 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-var-lib-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-device-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26d0251c-9cab-48df-bd56-a6724f77f293-hosts-file\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-var-lib-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-etc-kubernetes\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d0251c-9cab-48df-bd56-a6724f77f293-tmp-dir\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.716486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716301 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-slash\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72521405-0340-47a7-85e1-770cbe06b42e-iptables-alerter-script\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716343 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/26d0251c-9cab-48df-bd56-a6724f77f293-hosts-file\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdf96\" (UniqueName: \"kubernetes.io/projected/72521405-0340-47a7-85e1-770cbe06b42e-kube-api-access-sdf96\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716376 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-systemd\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysctl-d\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-sys\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-cnibin\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-cni-bin\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysctl-conf\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-sys\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716611 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-cni-bin\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-systemd\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d0251c-9cab-48df-bd56-a6724f77f293-tmp-dir\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716672 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.717343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e1410e4-456b-4092-afc8-e850a944a16a-ovn-node-metrics-cert\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716759 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysctl-conf\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-ovnkube-script-lib\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysctl-d\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716801 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-cnibin\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fa850c7-5397-4be3-80a9-2afab7eca821-kubelet-config\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/563c20c6-a7e1-412b-8682-95009c41620c-multus-daemon-config\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716915 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716927 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-kubelet\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-env-overrides\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-cni-bin\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-registration-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysconfig\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/563c20c6-a7e1-412b-8682-95009c41620c-cni-binary-copy\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8vl\" (UniqueName: \"kubernetes.io/projected/a1aa96c3-14fa-4594-ba74-97b26ff67379-kube-api-access-vp8vl\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.718180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-cni-netd\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717099 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-registration-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-sysconfig\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7fa850c7-5397-4be3-80a9-2afab7eca821-kubelet-config\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-ovnkube-config\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717197 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717206 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72521405-0340-47a7-85e1-770cbe06b42e-iptables-alerter-script\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717205 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-cni-netd\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716750 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-cni-bin\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.716876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-etc-selinux\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-kubelet\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717643 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-env-overrides\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717700 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-lib-modules\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717733 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/563c20c6-a7e1-412b-8682-95009c41620c-multus-daemon-config\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-cnibin\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717780 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-netns\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717789 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-lib-modules\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-conf-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717862 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-cnibin\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-netns\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717923 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-conf-dir\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-log-socket\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.717986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-socket-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-modprobe-d\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-log-socket\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-var-lib-kubelet\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-var-lib-kubelet\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-socket-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718166 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-modprobe-d\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7e1410e4-456b-4092-afc8-e850a944a16a-ovnkube-script-lib\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-socket-dir-parent\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-kubelet\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.719821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8h8x\" (UniqueName: \"kubernetes.io/projected/563c20c6-a7e1-412b-8682-95009c41620c-kube-api-access-x8h8x\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-multus-socket-dir-parent\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718313 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-kubelet\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1aa96c3-14fa-4594-ba74-97b26ff67379-serviceca\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-k8s-cni-cncf-io\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-cni-multus\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-multus-certs\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72521405-0340-47a7-85e1-770cbe06b42e-host-slash\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-k8s-cni-cncf-io\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-systemd-units\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/87136eb6-b040-4611-9150-8308af770a4b-konnectivity-ca\") pod \"konnectivity-agent-2m7lm\" (UID: \"87136eb6-b040-4611-9150-8308af770a4b\") " pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-run-multus-certs\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718502 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-host-var-lib-cni-multus\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-run-netns\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718539 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-systemd-units\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718541 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72521405-0340-47a7-85e1-770cbe06b42e-host-slash\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n52n4\" (UniqueName: \"kubernetes.io/projected/7e1410e4-456b-4092-afc8-e850a944a16a-kube-api-access-n52n4\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.720692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718586 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gsmb\" (UniqueName: \"kubernetes.io/projected/26d0251c-9cab-48df-bd56-a6724f77f293-kube-api-access-2gsmb\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-hostroot\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a1aa96c3-14fa-4594-ba74-97b26ff67379-serviceca\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-etc-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718724 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-host-run-netns\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-sys-fs\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/563c20c6-a7e1-412b-8682-95009c41620c-hostroot\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718774 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf5t\" (UniqueName: \"kubernetes.io/projected/1b325953-2f74-4fda-bc5b-cda5dc5d7290-kube-api-access-rlf5t\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-kubernetes\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-systemd\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/907eeeb1-e9b3-49df-a466-7eadf1141ae4-tmp\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-system-cni-dir\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-ovn\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-node-log\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.721695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.718986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-run\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-host\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjdtr\" (UniqueName: \"kubernetes.io/projected/907eeeb1-e9b3-49df-a466-7eadf1141ae4-kube-api-access-mjdtr\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719098 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/048e8329-979a-4219-8b27-c2f7a41ed82d-system-cni-dir\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719386 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-node-log\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-run\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719432 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-run-ovn\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-host\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719518 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-kubernetes\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-systemd\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/87136eb6-b040-4611-9150-8308af770a4b-konnectivity-ca\") pod \"konnectivity-agent-2m7lm\" (UID: \"87136eb6-b040-4611-9150-8308af770a4b\") " pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.719644 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719662 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7e1410e4-456b-4092-afc8-e850a944a16a-etc-openvswitch\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1b325953-2f74-4fda-bc5b-cda5dc5d7290-sys-fs\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.719727 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:44.219708401 +0000 UTC m=+2.073617072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:43.722635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.719977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/048e8329-979a-4219-8b27-c2f7a41ed82d-cni-binary-copy\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.723543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.720609 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/907eeeb1-e9b3-49df-a466-7eadf1141ae4-etc-tuned\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.723543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.720688 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7e1410e4-456b-4092-afc8-e850a944a16a-ovn-node-metrics-cert\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.723543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.720829 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/87136eb6-b040-4611-9150-8308af770a4b-agent-certs\") pod \"konnectivity-agent-2m7lm\" (UID: \"87136eb6-b040-4611-9150-8308af770a4b\") " pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.723543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.721243 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/907eeeb1-e9b3-49df-a466-7eadf1141ae4-tmp\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.726903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.726856 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnrkb\" (UniqueName: \"kubernetes.io/projected/048e8329-979a-4219-8b27-c2f7a41ed82d-kube-api-access-vnrkb\") pod \"multus-additional-cni-plugins-pv52q\" (UID: \"048e8329-979a-4219-8b27-c2f7a41ed82d\") " pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.727366 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.727347 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:19:43.727442 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.727375 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:19:43.727442 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.727390 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:43.727562 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:43.727452 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:44.227434925 +0000 UTC m=+2.081343601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:43.728807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.728763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcf8z\" (UniqueName: \"kubernetes.io/projected/16991ff6-9847-4586-b642-2db22ae33842-kube-api-access-rcf8z\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:43.729091 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.728955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf5t\" (UniqueName: \"kubernetes.io/projected/1b325953-2f74-4fda-bc5b-cda5dc5d7290-kube-api-access-rlf5t\") pod \"aws-ebs-csi-driver-node-sbpv7\" (UID: \"1b325953-2f74-4fda-bc5b-cda5dc5d7290\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.729168 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.729105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdf96\" (UniqueName: \"kubernetes.io/projected/72521405-0340-47a7-85e1-770cbe06b42e-kube-api-access-sdf96\") pod \"iptables-alerter-kfgv6\" (UID: \"72521405-0340-47a7-85e1-770cbe06b42e\") " pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.729233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.729164 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8vl\" (UniqueName: \"kubernetes.io/projected/a1aa96c3-14fa-4594-ba74-97b26ff67379-kube-api-access-vp8vl\") pod \"node-ca-pwhpf\" (UID: \"a1aa96c3-14fa-4594-ba74-97b26ff67379\") " pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.729447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.729425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8h8x\" (UniqueName: \"kubernetes.io/projected/563c20c6-a7e1-412b-8682-95009c41620c-kube-api-access-x8h8x\") pod \"multus-hmvj6\" (UID: \"563c20c6-a7e1-412b-8682-95009c41620c\") " pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.730304 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.730279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n52n4\" (UniqueName: \"kubernetes.io/projected/7e1410e4-456b-4092-afc8-e850a944a16a-kube-api-access-n52n4\") pod \"ovnkube-node-jrjvz\" (UID: \"7e1410e4-456b-4092-afc8-e850a944a16a\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.730457 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.730341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjdtr\" (UniqueName: \"kubernetes.io/projected/907eeeb1-e9b3-49df-a466-7eadf1141ae4-kube-api-access-mjdtr\") pod \"tuned-q4qfg\" (UID: \"907eeeb1-e9b3-49df-a466-7eadf1141ae4\") " pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.733203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.733179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gsmb\" (UniqueName: \"kubernetes.io/projected/26d0251c-9cab-48df-bd56-a6724f77f293-kube-api-access-2gsmb\") pod \"node-resolver-8998p\" (UID: \"26d0251c-9cab-48df-bd56-a6724f77f293\") " pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.832790 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.832727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hmvj6" Sep 10 14:19:43.853990 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.853965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-kfgv6" Sep 10 14:19:43.872285 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.872263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:19:43.889117 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.889098 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:19:43.909777 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.909754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" Sep 10 14:19:43.915626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.915608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" Sep 10 14:19:43.936288 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.936272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8998p" Sep 10 14:19:43.944609 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.944558 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e1410e4_456b_4092_afc8_e850a944a16a.slice/crio-56187d30d5ce599630f7817f8166fafd40f6277437e4e755e0fa10b29860bf1e WatchSource:0}: Error finding container 56187d30d5ce599630f7817f8166fafd40f6277437e4e755e0fa10b29860bf1e: Status 404 returned error can't find the container with id 56187d30d5ce599630f7817f8166fafd40f6277437e4e755e0fa10b29860bf1e Sep 10 14:19:43.944833 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.944798 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b325953_2f74_4fda_bc5b_cda5dc5d7290.slice/crio-92e91958c2dd2ed6337f3d61ea7c9222ca2185c5d408d8d681868c839b787207 WatchSource:0}: Error finding container 92e91958c2dd2ed6337f3d61ea7c9222ca2185c5d408d8d681868c839b787207: Status 404 returned error can't find the container with id 92e91958c2dd2ed6337f3d61ea7c9222ca2185c5d408d8d681868c839b787207 Sep 10 14:19:43.945307 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.945282 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87136eb6_b040_4611_9150_8308af770a4b.slice/crio-2964c81e55a590706d5ed17e20593a1fd560baf1fd0f95b22d631cf5c3e0b1a2 WatchSource:0}: Error finding container 2964c81e55a590706d5ed17e20593a1fd560baf1fd0f95b22d631cf5c3e0b1a2: Status 404 returned error can't find the container with id 2964c81e55a590706d5ed17e20593a1fd560baf1fd0f95b22d631cf5c3e0b1a2 Sep 10 14:19:43.945988 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.945960 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7a76a58af3f5a735a46f27cbfd95e9.slice/crio-03c5d744f10dbb2c2e29ca579d082596bda912a38c7d9a8432c427aec5d56f1b WatchSource:0}: Error finding container 03c5d744f10dbb2c2e29ca579d082596bda912a38c7d9a8432c427aec5d56f1b: Status 404 returned error can't find the container with id 03c5d744f10dbb2c2e29ca579d082596bda912a38c7d9a8432c427aec5d56f1b Sep 10 14:19:43.946548 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.946499 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e1c97d5f33b9f41328e5f5e35a0ce0.slice/crio-584981f46da6c47abfc5c528b22a52795219cbb2e1e16983af6363fd82d9b694 WatchSource:0}: Error finding container 584981f46da6c47abfc5c528b22a52795219cbb2e1e16983af6363fd82d9b694: Status 404 returned error can't find the container with id 584981f46da6c47abfc5c528b22a52795219cbb2e1e16983af6363fd82d9b694 Sep 10 14:19:43.948463 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.948321 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907eeeb1_e9b3_49df_a466_7eadf1141ae4.slice/crio-256897d3d28f4cb3825bac0cae5973ea217bf6c80f28a824686d6fb19b1b47f6 WatchSource:0}: Error finding container 256897d3d28f4cb3825bac0cae5973ea217bf6c80f28a824686d6fb19b1b47f6: Status 404 returned error can't find the container with id 256897d3d28f4cb3825bac0cae5973ea217bf6c80f28a824686d6fb19b1b47f6 Sep 10 14:19:43.953234 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.953115 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:19:43.953234 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.953179 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d0251c_9cab_48df_bd56_a6724f77f293.slice/crio-133e8bdf596f978882658f54e2c94d0e0904a8a7fa54a8fc332267ba64ece289 WatchSource:0}: Error finding container 133e8bdf596f978882658f54e2c94d0e0904a8a7fa54a8fc332267ba64ece289: Status 404 returned error can't find the container with id 133e8bdf596f978882658f54e2c94d0e0904a8a7fa54a8fc332267ba64ece289 Sep 10 14:19:43.962669 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.962651 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-pwhpf" Sep 10 14:19:43.967520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:43.967486 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pv52q" Sep 10 14:19:43.970560 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.970539 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1aa96c3_14fa_4594_ba74_97b26ff67379.slice/crio-a1804cad198dd33180507cb136f9da6cc6d5f541275bbce5cf5806b18727c6cd WatchSource:0}: Error finding container a1804cad198dd33180507cb136f9da6cc6d5f541275bbce5cf5806b18727c6cd: Status 404 returned error can't find the container with id a1804cad198dd33180507cb136f9da6cc6d5f541275bbce5cf5806b18727c6cd Sep 10 14:19:43.975345 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:43.975323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048e8329_979a_4219_8b27_c2f7a41ed82d.slice/crio-2cf7cb1c19e7b009a6f17ee244270055f8299a4dfc03e8c29188eaf07d106932 WatchSource:0}: Error finding container 2cf7cb1c19e7b009a6f17ee244270055f8299a4dfc03e8c29188eaf07d106932: Status 404 returned error can't find the container with id 2cf7cb1c19e7b009a6f17ee244270055f8299a4dfc03e8c29188eaf07d106932 Sep 10 14:19:44.000176 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.000145 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:19:44.169878 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:44.169846 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563c20c6_a7e1_412b_8682_95009c41620c.slice/crio-31b1e755555bc15678824fc94d8d331cfc7e5a6f3fe3893a952cb482dd41e84c WatchSource:0}: Error finding container 31b1e755555bc15678824fc94d8d331cfc7e5a6f3fe3893a952cb482dd41e84c: Status 404 returned error can't find the container with id 31b1e755555bc15678824fc94d8d331cfc7e5a6f3fe3893a952cb482dd41e84c Sep 10 14:19:44.206005 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:19:44.205982 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72521405_0340_47a7_85e1_770cbe06b42e.slice/crio-3f70611723744d79c7aeb901cda530b53f8ede8901f8cdd6202ef6c58869bb26 WatchSource:0}: Error finding container 3f70611723744d79c7aeb901cda530b53f8ede8901f8cdd6202ef6c58869bb26: Status 404 returned error can't find the container with id 3f70611723744d79c7aeb901cda530b53f8ede8901f8cdd6202ef6c58869bb26 Sep 10 14:19:44.222868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.222842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:44.222951 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.222897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:44.222992 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.222986 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:44.223035 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.222988 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:44.223035 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.223033 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:45.223019014 +0000 UTC m=+3.076927669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:44.223114 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.223046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:45.223040465 +0000 UTC m=+3.076949120 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:44.324069 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.324042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:44.324186 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.324164 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:19:44.324186 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.324184 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:19:44.324270 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.324193 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:44.324270 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.324242 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:45.324227696 +0000 UTC m=+3.178136366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:44.450809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.450722 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:19:44.608573 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.608502 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:14:44 +0000 UTC" deadline="2028-03-06 23:46:00.905577135 +0000 UTC" Sep 10 14:19:44.608573 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.608569 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13041h26m16.297012181s" Sep 10 14:19:44.757827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.757441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:44.757827 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:44.757802 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:44.772004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.771950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" event={"ID":"72e1c97d5f33b9f41328e5f5e35a0ce0","Type":"ContainerStarted","Data":"584981f46da6c47abfc5c528b22a52795219cbb2e1e16983af6363fd82d9b694"} Sep 10 14:19:44.793291 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.793221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2m7lm" event={"ID":"87136eb6-b040-4611-9150-8308af770a4b","Type":"ContainerStarted","Data":"2964c81e55a590706d5ed17e20593a1fd560baf1fd0f95b22d631cf5c3e0b1a2"} Sep 10 14:19:44.812952 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.812794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kfgv6" event={"ID":"72521405-0340-47a7-85e1-770cbe06b42e","Type":"ContainerStarted","Data":"3f70611723744d79c7aeb901cda530b53f8ede8901f8cdd6202ef6c58869bb26"} Sep 10 14:19:44.838310 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.838271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hmvj6" event={"ID":"563c20c6-a7e1-412b-8682-95009c41620c","Type":"ContainerStarted","Data":"31b1e755555bc15678824fc94d8d331cfc7e5a6f3fe3893a952cb482dd41e84c"} Sep 10 14:19:44.858898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.857473 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pwhpf" event={"ID":"a1aa96c3-14fa-4594-ba74-97b26ff67379","Type":"ContainerStarted","Data":"a1804cad198dd33180507cb136f9da6cc6d5f541275bbce5cf5806b18727c6cd"} Sep 10 14:19:44.873573 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.873539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8998p" event={"ID":"26d0251c-9cab-48df-bd56-a6724f77f293","Type":"ContainerStarted","Data":"133e8bdf596f978882658f54e2c94d0e0904a8a7fa54a8fc332267ba64ece289"} Sep 10 14:19:44.880910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.880849 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" event={"ID":"907eeeb1-e9b3-49df-a466-7eadf1141ae4","Type":"ContainerStarted","Data":"256897d3d28f4cb3825bac0cae5973ea217bf6c80f28a824686d6fb19b1b47f6"} Sep 10 14:19:44.892501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.892472 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"56187d30d5ce599630f7817f8166fafd40f6277437e4e755e0fa10b29860bf1e"} Sep 10 14:19:44.902584 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.902498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" event={"ID":"7a7a76a58af3f5a735a46f27cbfd95e9","Type":"ContainerStarted","Data":"03c5d744f10dbb2c2e29ca579d082596bda912a38c7d9a8432c427aec5d56f1b"} Sep 10 14:19:44.917040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.917000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerStarted","Data":"2cf7cb1c19e7b009a6f17ee244270055f8299a4dfc03e8c29188eaf07d106932"} Sep 10 14:19:44.933944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:44.933679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" event={"ID":"1b325953-2f74-4fda-bc5b-cda5dc5d7290","Type":"ContainerStarted","Data":"92e91958c2dd2ed6337f3d61ea7c9222ca2185c5d408d8d681868c839b787207"} Sep 10 14:19:45.234796 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.234615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:45.234796 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.234697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:45.235087 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.234836 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:45.235087 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.234922 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:47.234901243 +0000 UTC m=+5.088809914 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:45.235087 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.235007 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:45.235087 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.235046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:47.235032529 +0000 UTC m=+5.088941200 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:45.336109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.336052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:45.336317 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.336250 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:19:45.336317 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.336276 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:19:45.336317 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.336290 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:45.336485 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.336354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:47.336334818 +0000 UTC m=+5.190243489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:45.452040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.452007 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 14:19:45.609540 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.609453 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 14:14:44 +0000 UTC" deadline="2028-03-20 17:12:15.081861577 +0000 UTC" Sep 10 14:19:45.609540 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.609494 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13370h52m29.472374394s" Sep 10 14:19:45.755041 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.755005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:45.755237 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.755186 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:45.755298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:45.755268 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:45.755426 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:45.755403 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:46.756045 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:46.755994 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:46.756531 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:46.756144 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:47.254302 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:47.254211 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:47.254302 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:47.254295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:47.254538 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.254428 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:47.254538 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.254492 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.254473113 +0000 UTC m=+9.108381785 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:47.254943 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.254921 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:47.255036 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.254987 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.254967529 +0000 UTC m=+9.108876445 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:47.355687 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:47.355644 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:47.355891 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.355817 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:19:47.355891 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.355858 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:19:47.355891 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.355871 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:47.356081 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.355945 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:51.355924333 +0000 UTC m=+9.209833008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:47.755830 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:47.755090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:47.755830 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.755231 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:47.755830 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:47.755665 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:47.755830 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:47.755777 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:48.757548 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:48.757035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:48.757548 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:48.757192 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:49.755178 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:49.755144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:49.755352 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:49.755281 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:49.755438 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:49.755351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:49.755493 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:49.755435 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:50.755177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:50.755142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:50.755711 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:50.755286 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:51.287348 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:51.287304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:51.287552 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:51.287397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:51.287552 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.287492 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:51.287692 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.287555 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:51.287692 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.287584 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:59.287560646 +0000 UTC m=+17.141469319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:51.287692 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.287610 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:59.28759324 +0000 UTC m=+17.141501904 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:51.388936 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:51.388898 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:51.389133 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.389110 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:19:51.389216 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.389143 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:19:51.389216 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.389159 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:51.389306 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.389229 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:19:59.389208453 +0000 UTC m=+17.243117123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:51.754246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:51.754199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:51.754427 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.754344 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:51.754756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:51.754733 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:51.754864 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:51.754839 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:52.755720 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:52.755686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:52.756205 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:52.755823 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:53.755209 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:53.755125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:53.755395 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:53.755136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:53.755395 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:53.755269 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:53.755395 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:53.755332 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:54.755018 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:54.754985 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:54.755458 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:54.755120 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:55.755198 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:55.755161 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:55.755628 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:55.755276 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:55.755628 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:55.755332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:55.755628 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:55.755440 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:56.754751 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:56.754724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:56.754942 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:56.754847 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:57.754995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:57.754964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:57.755376 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:57.754967 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:57.755376 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:57.755063 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:19:57.755376 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:57.755179 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:58.754822 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:58.754779 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:58.754989 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:58.754916 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:19:59.349372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:59.349332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:59.349903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:59.349394 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:59.349903 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.349487 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:59.349903 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.349553 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:15.349536991 +0000 UTC m=+33.203445644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:19:59.349903 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.349495 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:59.349903 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.349634 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:15.349617545 +0000 UTC m=+33.203526209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:19:59.449738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:59.449695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:19:59.449905 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.449889 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:19:59.450006 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.449912 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:19:59.450006 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.449925 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:59.450006 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.449981 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:15.449967788 +0000 UTC m=+33.303876456 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:19:59.754756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:59.754723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:19:59.754756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:19:59.754749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:19:59.754949 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.754848 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:19:59.754986 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:19:59.754959 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:00.754807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:00.754761 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:00.755261 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:00.754899 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:01.754956 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:01.754917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:01.755367 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:01.754917 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:01.755367 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:01.755063 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:01.755367 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:01.755139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:02.755352 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:02.755315 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:02.755839 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:02.755424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:03.754873 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:03.754848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:03.754873 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:03.754869 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:03.755103 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:03.754979 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:03.755103 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:03.755095 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:03.978280 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:03.978117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" event={"ID":"907eeeb1-e9b3-49df-a466-7eadf1141ae4","Type":"ContainerStarted","Data":"db10b18c9e0ae12debc3a6d279807b7ade51137ccda7a9235808756ff209416a"} Sep 10 14:20:03.979554 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:03.979533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"905fe3a2f261eec6385f15027f68ba81536c375441ed1457632924dde14549e5"} Sep 10 14:20:03.980935 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:03.980916 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" event={"ID":"72e1c97d5f33b9f41328e5f5e35a0ce0","Type":"ContainerStarted","Data":"6f34d5a9a63a4bbd5b4c2640322e0a9533c1b8586d188298de9c60345ffd56f5"} Sep 10 14:20:03.995453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:03.995406 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q4qfg" podStartSLOduration=1.233388573 podStartE2EDuration="20.995389867s" totalImagesPullingTime="19.762001294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:03.99474982 +0000 UTC m=+21.848658521" watchObservedRunningTime="2026-09-10 14:20:03.995389867 +0000 UTC m=+21.849298544" Sep 10 14:20:04.754683 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.754524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:04.754827 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:04.754754 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:04.984932 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.984896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-pwhpf" event={"ID":"a1aa96c3-14fa-4594-ba74-97b26ff67379","Type":"ContainerStarted","Data":"b047c70ca0e04614c3a320f49abab3e01e48d8986aa46082382bcb50daa0c4ac"} Sep 10 14:20:04.986610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.986587 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8998p" event={"ID":"26d0251c-9cab-48df-bd56-a6724f77f293","Type":"ContainerStarted","Data":"c6c87fed984db642df52a1bf2604976a135f91adb3ba298f1ee110692e6a225d"} Sep 10 14:20:04.989961 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.989913 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:20:04.990313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.990294 2576 generic.go:356] "Generic (PLEG): container finished" podID="7e1410e4-456b-4092-afc8-e850a944a16a" containerID="ac7ff81f51c2c8658670b7e27fb83199e52feb2c8927645f1925515eb56286f0" exitCode=1 Sep 10 14:20:04.990409 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.990336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"20472e4cdc07f3b687dfe7172853b9d10c5005b8992b0998d65316559d58f885"} Sep 10 14:20:04.990473 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.990414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"35b5b3a2303aab42d399cb17492c3a84e58f0d0de37ef2a0cfc0539f3f956831"} Sep 10 14:20:04.990473 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.990468 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"43f492aee2b0964c1d226be01b242e9ec418712ce9f7fa9c8e36168270cf8152"} Sep 10 14:20:04.990553 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.990513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"0387cc18c23cbed87f3e40d81564b4b9d031d0f10a14278294b98c3a0ffdeda6"} Sep 10 14:20:04.990604 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.990561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerDied","Data":"ac7ff81f51c2c8658670b7e27fb83199e52feb2c8927645f1925515eb56286f0"} Sep 10 14:20:04.992014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.991992 2576 generic.go:356] "Generic (PLEG): container finished" podID="7a7a76a58af3f5a735a46f27cbfd95e9" containerID="e2a79f4458b9d7f45fa93ca33b0e078104dfdb579bce41ffcb8ca8f76df82c34" exitCode=0 Sep 10 14:20:04.992122 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.992069 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" event={"ID":"7a7a76a58af3f5a735a46f27cbfd95e9","Type":"ContainerDied","Data":"e2a79f4458b9d7f45fa93ca33b0e078104dfdb579bce41ffcb8ca8f76df82c34"} Sep 10 14:20:04.993814 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.993776 2576 generic.go:356] "Generic (PLEG): container finished" podID="048e8329-979a-4219-8b27-c2f7a41ed82d" containerID="34ca97ca89e9b00992fe72aeb78d04d2e666bb700aa1ebe1c759d4eefce0ddfb" exitCode=0 Sep 10 14:20:04.993905 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.993821 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerDied","Data":"34ca97ca89e9b00992fe72aeb78d04d2e666bb700aa1ebe1c759d4eefce0ddfb"} Sep 10 14:20:04.995847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.995820 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" event={"ID":"1b325953-2f74-4fda-bc5b-cda5dc5d7290","Type":"ContainerStarted","Data":"49abb56160accffb5285584d595a047961d19505968d3d4890091fb0f4711f94"} Sep 10 14:20:04.997497 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.997464 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2m7lm" event={"ID":"87136eb6-b040-4611-9150-8308af770a4b","Type":"ContainerStarted","Data":"87d983344ad429f1f95430bd7004d01772dd75536c1ebb8121fa1423b019bb6e"} Sep 10 14:20:04.999090 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:04.999037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-kfgv6" event={"ID":"72521405-0340-47a7-85e1-770cbe06b42e","Type":"ContainerStarted","Data":"94114abd5a10336c587d6cd3cda4f8e3a6b6f16e1abfc1d8b8e1fc096681869f"} Sep 10 14:20:05.001083 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.001036 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-236.ec2.internal" podStartSLOduration=21.001018479 podStartE2EDuration="21.001018479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:04.007889897 +0000 UTC m=+21.861798574" watchObservedRunningTime="2026-09-10 14:20:05.001018479 +0000 UTC m=+22.854927172" Sep 10 14:20:05.003420 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.003396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hmvj6" event={"ID":"563c20c6-a7e1-412b-8682-95009c41620c","Type":"ContainerStarted","Data":"4654693bcfd4e05347d9b2053c45787c15009423efbfeae798f9b93c373fbd96"} Sep 10 14:20:05.014519 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.014482 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-pwhpf" podStartSLOduration=2.306934965 podStartE2EDuration="22.014470529s" totalImagesPullingTime="19.707535564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:05.000595571 +0000 UTC m=+22.854504281" watchObservedRunningTime="2026-09-10 14:20:05.014470529 +0000 UTC m=+22.868379205" Sep 10 14:20:05.014853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.014825 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-kfgv6" podStartSLOduration=2.506802997 podStartE2EDuration="22.014816384s" totalImagesPullingTime="19.508013387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:05.014266476 +0000 UTC m=+22.868175153" watchObservedRunningTime="2026-09-10 14:20:05.014816384 +0000 UTC m=+22.868725058" Sep 10 14:20:05.067559 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.067507 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2m7lm" podStartSLOduration=2.340786384 podStartE2EDuration="22.067488181s" totalImagesPullingTime="19.726701797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:05.0669742 +0000 UTC m=+22.920882878" watchObservedRunningTime="2026-09-10 14:20:05.067488181 +0000 UTC m=+22.921396859" Sep 10 14:20:05.084696 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.084643 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8998p" podStartSLOduration=2.326279978 podStartE2EDuration="22.08462979s" totalImagesPullingTime="19.758349812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:05.084434912 +0000 UTC m=+22.938343588" watchObservedRunningTime="2026-09-10 14:20:05.08462979 +0000 UTC m=+22.938538465" Sep 10 14:20:05.101873 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.101702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:05.102003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.101967 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hmvj6" podStartSLOduration=2.180769753 podStartE2EDuration="22.101952652s" totalImagesPullingTime="19.921182899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:05.10182576 +0000 UTC m=+22.955734437" watchObservedRunningTime="2026-09-10 14:20:05.101952652 +0000 UTC m=+22.955861321" Sep 10 14:20:05.162147 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.162116 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:05.353057 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.352971 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:05.420783 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.420753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:05.755090 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.755066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:05.755236 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.755100 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:05.755236 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:05.755176 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:05.755347 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:05.755280 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:05.776765 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:05.776639 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 14:20:06.008487 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.008400 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" event={"ID":"1b325953-2f74-4fda-bc5b-cda5dc5d7290","Type":"ContainerStarted","Data":"23392a787b6fde7991a5b5b013fea2e722d8b943e4728a4526fa523df85489c4"} Sep 10 14:20:06.010552 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.010520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" event={"ID":"7a7a76a58af3f5a735a46f27cbfd95e9","Type":"ContainerStarted","Data":"79cbeb4fed38fece864445605365162fb37015bd51bdbe9bc54e4ec66062e44b"} Sep 10 14:20:06.026746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.026688 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-236.ec2.internal" podStartSLOduration=22.026668831 podStartE2EDuration="22.026668831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:06.025309612 +0000 UTC m=+23.879218288" watchObservedRunningTime="2026-09-10 14:20:06.026668831 +0000 UTC m=+23.880577510" Sep 10 14:20:06.156165 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.156091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:06.208799 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.208762 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:06.546854 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.546822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:06.560511 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.560412 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T14:20:05.776653347Z","UUID":"1ef5a5f9-6e51-4e91-832c-e3e6b354d4fb","Handler":null,"Name":"","Endpoint":""} Sep 10 14:20:06.562130 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.562111 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 14:20:06.562239 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.562144 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 14:20:06.647376 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.647297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:06.755456 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:06.754722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:06.755456 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:06.754863 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:07.016973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.016899 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:20:07.017382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.017355 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"8d3e4a862aef2646b2066cc9e6b2d880e152cb496b04104deae064497a668dad"} Sep 10 14:20:07.486578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.485530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:07.754844 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.754813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:07.755065 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.754813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:07.755065 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:07.755029 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:07.755065 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:07.755050 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:07.826967 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.826929 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:20:07.827656 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:07.827631 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:20:08.022778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:08.022686 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" event={"ID":"1b325953-2f74-4fda-bc5b-cda5dc5d7290","Type":"ContainerStarted","Data":"9caba25687f9670e249589028ce3ed981fe3bd62b479f0dccae3db5bab8401f2"} Sep 10 14:20:08.023197 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:08.023155 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:20:08.023631 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:08.023612 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2m7lm" Sep 10 14:20:08.041645 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:08.041594 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sbpv7" podStartSLOduration=2.563409463 podStartE2EDuration="25.041580019s" totalImagesPullingTime="22.478170556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:08.040953507 +0000 UTC m=+25.894862183" watchObservedRunningTime="2026-09-10 14:20:08.041580019 +0000 UTC m=+25.895488695" Sep 10 14:20:08.092449 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:08.092408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:08.754749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:08.754717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:08.754926 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:08.754870 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:09.754874 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:09.754773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:09.755275 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:09.754904 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:09.755275 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:09.754961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:09.755275 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:09.755094 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:10.030240 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.030168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:20:10.030578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.030552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"b897326bfe60c795c0db64ed44f241365cfe194b363bbe6e5ca1287aa18d559b"} Sep 10 14:20:10.030811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.030775 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:20:10.031004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.030987 2576 scope.go:117] "RemoveContainer" containerID="ac7ff81f51c2c8658670b7e27fb83199e52feb2c8927645f1925515eb56286f0" Sep 10 14:20:10.032591 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.032567 2576 generic.go:356] "Generic (PLEG): container finished" podID="048e8329-979a-4219-8b27-c2f7a41ed82d" containerID="0d8a460be3b5908045fbe9d4d46888511a14e1956d2746f63c1ac0e107160e31" exitCode=0 Sep 10 14:20:10.032673 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.032615 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerDied","Data":"0d8a460be3b5908045fbe9d4d46888511a14e1956d2746f63c1ac0e107160e31"} Sep 10 14:20:10.047975 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.047954 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:20:10.754776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:10.754755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:10.754889 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:10.754870 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:11.036807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.036761 2576 generic.go:356] "Generic (PLEG): container finished" podID="048e8329-979a-4219-8b27-c2f7a41ed82d" containerID="4934321e0b585aa42c2d106c78c44f599ee25e11e11a6f7a95f52d9b97444138" exitCode=0 Sep 10 14:20:11.036962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.036843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerDied","Data":"4934321e0b585aa42c2d106c78c44f599ee25e11e11a6f7a95f52d9b97444138"} Sep 10 14:20:11.040888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.040870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:20:11.041298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.041271 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" event={"ID":"7e1410e4-456b-4092-afc8-e850a944a16a","Type":"ContainerStarted","Data":"32bcb9519e38a3954a1ec2187aa88fcfd05c3c3b9b1826f5dfb1a17992baee51"} Sep 10 14:20:11.041557 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.041498 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:20:11.041630 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.041582 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:20:11.057341 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.057319 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:20:11.088592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.088547 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" podStartSLOduration=8.299199528 podStartE2EDuration="28.088534547s" totalImagesPullingTime="19.789335019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:11.088517566 +0000 UTC m=+28.942426254" watchObservedRunningTime="2026-09-10 14:20:11.088534547 +0000 UTC m=+28.942443204" Sep 10 14:20:11.147225 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.147051 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7h85z"] Sep 10 14:20:11.147359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.147346 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:11.147469 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:11.147449 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:11.150786 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.150765 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9pl2k"] Sep 10 14:20:11.150902 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.150875 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:11.150988 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:11.150966 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:11.151309 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.151286 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q996v"] Sep 10 14:20:11.151420 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:11.151390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:11.151508 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:11.151489 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:12.755287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:12.755264 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:12.755668 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:12.755347 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:12.755668 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:12.755383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:12.755668 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:12.755425 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:12.755668 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:12.755448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:12.755668 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:12.755489 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:13.048896 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:13.048815 2576 generic.go:356] "Generic (PLEG): container finished" podID="048e8329-979a-4219-8b27-c2f7a41ed82d" containerID="eeffc478a79a3ed0e27e7ab0030ee60e6249ad3c6a9cfa3d53375f2877066927" exitCode=0 Sep 10 14:20:13.049017 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:13.048892 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerDied","Data":"eeffc478a79a3ed0e27e7ab0030ee60e6249ad3c6a9cfa3d53375f2877066927"} Sep 10 14:20:14.033948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:14.033905 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:14.134988 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:14.134945 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:14.755200 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:14.755171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:14.755322 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:14.755218 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:14.755322 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:14.755171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:14.755426 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:14.755359 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:14.755494 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:14.755458 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:14.755647 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:14.755596 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:15.364928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:15.364900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:15.365376 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:15.364961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:15.365376 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.365056 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:20:15.365376 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.365092 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 14:20:15.365376 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.365125 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs podName:16991ff6-9847-4586-b642-2db22ae33842 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.365110624 +0000 UTC m=+65.219019282 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs") pod "network-metrics-daemon-q996v" (UID: "16991ff6-9847-4586-b642-2db22ae33842") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 14:20:15.365376 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.365142 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret podName:7fa850c7-5397-4be3-80a9-2afab7eca821 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.365135471 +0000 UTC m=+65.219044126 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret") pod "global-pull-secret-syncer-9pl2k" (UID: "7fa850c7-5397-4be3-80a9-2afab7eca821") : object "kube-system"/"original-pull-secret" not registered Sep 10 14:20:15.465911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:15.465874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:15.466071 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.466048 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 14:20:15.466116 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.466080 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 14:20:15.466116 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.466094 2576 projected.go:196] Error preparing data for projected volume kube-api-access-x2vrr for pod openshift-network-diagnostics/network-check-target-7h85z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:20:15.466197 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:15.466161 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr podName:8f84440f-6fb4-469c-bc1b-ab69e65c87e2 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:47.466142905 +0000 UTC m=+65.320051582 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x2vrr" (UniqueName: "kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr") pod "network-check-target-7h85z" (UID: "8f84440f-6fb4-469c-bc1b-ab69e65c87e2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 14:20:16.256836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.256787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:16.355610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.355535 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:16.755401 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.755364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:16.755847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.755364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:16.755847 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:16.755510 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7h85z" podUID="8f84440f-6fb4-469c-bc1b-ab69e65c87e2" Sep 10 14:20:16.755847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.755536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:16.755847 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:16.755651 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q996v" podUID="16991ff6-9847-4586-b642-2db22ae33842" Sep 10 14:20:16.755847 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:16.755728 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9pl2k" podUID="7fa850c7-5397-4be3-80a9-2afab7eca821" Sep 10 14:20:16.923247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.923164 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-236.ec2.internal" event="NodeReady" Sep 10 14:20:16.923485 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.923335 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 14:20:16.975461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.975423 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gb9tp"] Sep 10 14:20:16.995770 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.995739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-25qhx"] Sep 10 14:20:16.995995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.995885 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:16.998891 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.998866 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 14:20:16.999034 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.998866 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 14:20:16.999220 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.999200 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 14:20:16.999422 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:16.999406 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tsxlz\"" Sep 10 14:20:17.007807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.007771 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25qhx"] Sep 10 14:20:17.007913 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.007824 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gb9tp"] Sep 10 14:20:17.007913 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.007908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.010657 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.010624 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 14:20:17.010841 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.010824 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 14:20:17.010841 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.010838 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bskzl\"" Sep 10 14:20:17.114047 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.114004 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:17.170090 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.170029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:17.180694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.180625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/649a3eab-a879-4a4b-9d33-5e3d621a5953-config-volume\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.180694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.180657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.180694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.180679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649a3eab-a879-4a4b-9d33-5e3d621a5953-tmp-dir\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.180949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.180815 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40-cert\") pod \"ingress-canary-gb9tp\" (UID: \"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40\") " pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.180949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.180894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjf2g\" (UniqueName: \"kubernetes.io/projected/5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40-kube-api-access-gjf2g\") pod \"ingress-canary-gb9tp\" (UID: \"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40\") " pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.180949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.180927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmglt\" (UniqueName: \"kubernetes.io/projected/649a3eab-a879-4a4b-9d33-5e3d621a5953-kube-api-access-qmglt\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.282222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjf2g\" (UniqueName: \"kubernetes.io/projected/5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40-kube-api-access-gjf2g\") pod \"ingress-canary-gb9tp\" (UID: \"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40\") " pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.282503 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmglt\" (UniqueName: \"kubernetes.io/projected/649a3eab-a879-4a4b-9d33-5e3d621a5953-kube-api-access-qmglt\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.282503 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/649a3eab-a879-4a4b-9d33-5e3d621a5953-config-volume\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.282503 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.282503 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649a3eab-a879-4a4b-9d33-5e3d621a5953-tmp-dir\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.282503 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40-cert\") pod \"ingress-canary-gb9tp\" (UID: \"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40\") " pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.282771 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:17.282526 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:20:17.282771 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:17.282591 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls podName:649a3eab-a879-4a4b-9d33-5e3d621a5953 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:17.78257104 +0000 UTC m=+35.636479710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls") pod "dns-default-25qhx" (UID: "649a3eab-a879-4a4b-9d33-5e3d621a5953") : secret "dns-default-metrics-tls" not found Sep 10 14:20:17.282909 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282874 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/649a3eab-a879-4a4b-9d33-5e3d621a5953-tmp-dir\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.282990 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.282970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/649a3eab-a879-4a4b-9d33-5e3d621a5953-config-volume\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.287561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.287433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40-cert\") pod \"ingress-canary-gb9tp\" (UID: \"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40\") " pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.292744 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.292723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjf2g\" (UniqueName: \"kubernetes.io/projected/5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40-kube-api-access-gjf2g\") pod \"ingress-canary-gb9tp\" (UID: \"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40\") " pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.293095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.293075 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmglt\" (UniqueName: \"kubernetes.io/projected/649a3eab-a879-4a4b-9d33-5e3d621a5953-kube-api-access-qmglt\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.307020 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.306996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gb9tp" Sep 10 14:20:17.599571 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.599543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:17.700217 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.700172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:17.775316 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.775292 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gb9tp"] Sep 10 14:20:17.786658 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:17.786638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:17.786825 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:17.786782 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:20:17.786889 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:17.786878 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls podName:649a3eab-a879-4a4b-9d33-5e3d621a5953 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:18.786863531 +0000 UTC m=+36.640772184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls") pod "dns-default-25qhx" (UID: "649a3eab-a879-4a4b-9d33-5e3d621a5953") : secret "dns-default-metrics-tls" not found Sep 10 14:20:17.787749 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:17.787729 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8bb0f6_c6b4_4a22_b678_8f2b65d28b40.slice/crio-d1d65205af642613ac34005ebb75e16e19aa56bb192d21f34046b2f82ea28623 WatchSource:0}: Error finding container d1d65205af642613ac34005ebb75e16e19aa56bb192d21f34046b2f82ea28623: Status 404 returned error can't find the container with id d1d65205af642613ac34005ebb75e16e19aa56bb192d21f34046b2f82ea28623 Sep 10 14:20:18.061097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.061019 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gb9tp" event={"ID":"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40","Type":"ContainerStarted","Data":"d1d65205af642613ac34005ebb75e16e19aa56bb192d21f34046b2f82ea28623"} Sep 10 14:20:18.063729 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.063702 2576 generic.go:356] "Generic (PLEG): container finished" podID="048e8329-979a-4219-8b27-c2f7a41ed82d" containerID="5852dad19873207c28a25836f12ff2cb435de30fa1bda2c749c0a1e8e334d848" exitCode=0 Sep 10 14:20:18.063875 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.063740 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerDied","Data":"5852dad19873207c28a25836f12ff2cb435de30fa1bda2c749c0a1e8e334d848"} Sep 10 14:20:18.755176 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.755137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:18.755345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.755137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:18.755345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.755277 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:18.758050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.758029 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 14:20:18.758421 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.758400 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 14:20:18.758421 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.758413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 14:20:18.759674 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.759646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 14:20:18.759674 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.759657 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q47dh\"" Sep 10 14:20:18.759819 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.759686 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5vvv\"" Sep 10 14:20:18.794780 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:18.794750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:18.795192 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:18.794893 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:20:18.795192 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:18.794958 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls podName:649a3eab-a879-4a4b-9d33-5e3d621a5953 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:20.794943064 +0000 UTC m=+38.648851718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls") pod "dns-default-25qhx" (UID: "649a3eab-a879-4a4b-9d33-5e3d621a5953") : secret "dns-default-metrics-tls" not found Sep 10 14:20:19.069852 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:19.069762 2576 generic.go:356] "Generic (PLEG): container finished" podID="048e8329-979a-4219-8b27-c2f7a41ed82d" containerID="6e3e310d54dc8495763c73634d52c0caa66239b2c924706f4c6e4a1c75b1d27e" exitCode=0 Sep 10 14:20:19.069852 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:19.069828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerDied","Data":"6e3e310d54dc8495763c73634d52c0caa66239b2c924706f4c6e4a1c75b1d27e"} Sep 10 14:20:20.075141 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.074956 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pv52q" event={"ID":"048e8329-979a-4219-8b27-c2f7a41ed82d","Type":"ContainerStarted","Data":"efd9991b6d63e083c21327d65c6e799f0c38346c2fe1ac6c5ec67dfff4f3e737"} Sep 10 14:20:20.098128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.097561 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pv52q" podStartSLOduration=6.538085795 podStartE2EDuration="37.09753847s" totalImagesPullingTime="30.559452675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:20.095753339 +0000 UTC m=+37.949662019" watchObservedRunningTime="2026-09-10 14:20:20.09753847 +0000 UTC m=+37.951447151" Sep 10 14:20:20.211636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.211605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:20.302423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.302385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:20.406583 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.406529 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:20.810999 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.810967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:20.811143 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:20.811124 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:20:20.811201 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:20.811189 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls podName:649a3eab-a879-4a4b-9d33-5e3d621a5953 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:24.811174244 +0000 UTC m=+42.665082908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls") pod "dns-default-25qhx" (UID: "649a3eab-a879-4a4b-9d33-5e3d621a5953") : secret "dns-default-metrics-tls" not found Sep 10 14:20:20.898842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.898782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:20.938469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.938398 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh"] Sep 10 14:20:20.954453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.954421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh"] Sep 10 14:20:20.954599 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.954553 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" Sep 10 14:20:20.957270 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.957244 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-gfspz\"" Sep 10 14:20:20.957455 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.957437 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 14:20:20.957539 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:20.957503 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 14:20:21.005573 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.005531 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:21.040869 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.040841 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k5ngq"] Sep 10 14:20:21.047215 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.047190 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.050107 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.050084 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 14:20:21.050235 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.050134 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 14:20:21.050369 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.050347 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 14:20:21.050448 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.050375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mdxkp\"" Sep 10 14:20:21.050627 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.050609 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 14:20:21.056533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.056512 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k5ngq"] Sep 10 14:20:21.078967 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.078934 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gb9tp" event={"ID":"5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40","Type":"ContainerStarted","Data":"9f2cc8d6b69024a68a44abac00b8bd01acaada47af9e0f7ea90c21db5f829b10"} Sep 10 14:20:21.093591 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.093546 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gb9tp" podStartSLOduration=2.7721921959999998 podStartE2EDuration="5.093531826s" totalImagesPullingTime="2.32133963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:21.093022413 +0000 UTC m=+38.946931090" watchObservedRunningTime="2026-09-10 14:20:21.093531826 +0000 UTC m=+38.947440501" Sep 10 14:20:21.097506 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.097443 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:21.113222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.113191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj5wj\" (UniqueName: \"kubernetes.io/projected/c0fb0680-1595-43fa-ac3d-c338139b3eb4-kube-api-access-tj5wj\") pod \"migrator-fdbb8fdff-m7qjh\" (UID: \"c0fb0680-1595-43fa-ac3d-c338139b3eb4\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" Sep 10 14:20:21.214081 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.213987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca9f5f01-a250-4c9c-9d20-6bf062b43632-crio-socket\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.214253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.214114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.214253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.214177 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca9f5f01-a250-4c9c-9d20-6bf062b43632-data-volume\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.214253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.214223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.214423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.214352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwl7\" (UniqueName: \"kubernetes.io/projected/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-api-access-8gwl7\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.214481 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.214430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj5wj\" (UniqueName: \"kubernetes.io/projected/c0fb0680-1595-43fa-ac3d-c338139b3eb4-kube-api-access-tj5wj\") pod \"migrator-fdbb8fdff-m7qjh\" (UID: \"c0fb0680-1595-43fa-ac3d-c338139b3eb4\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" Sep 10 14:20:21.226304 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.226276 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj5wj\" (UniqueName: \"kubernetes.io/projected/c0fb0680-1595-43fa-ac3d-c338139b3eb4-kube-api-access-tj5wj\") pod \"migrator-fdbb8fdff-m7qjh\" (UID: \"c0fb0680-1595-43fa-ac3d-c338139b3eb4\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" Sep 10 14:20:21.263125 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.263093 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" Sep 10 14:20:21.315439 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca9f5f01-a250-4c9c-9d20-6bf062b43632-crio-socket\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.315610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315481 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.315610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca9f5f01-a250-4c9c-9d20-6bf062b43632-data-volume\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.315610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.315610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwl7\" (UniqueName: \"kubernetes.io/projected/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-api-access-8gwl7\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.315753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ca9f5f01-a250-4c9c-9d20-6bf062b43632-crio-socket\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.315753 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.315663 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:20:21.315852 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.315774 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.815752385 +0000 UTC m=+39.669661063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : configmap "kube-rbac-proxy" not found Sep 10 14:20:21.315852 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.315817 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:20:21.315958 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.315894 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:21.815874628 +0000 UTC m=+39.669783284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : secret "insights-runtime-extractor-tls" not found Sep 10 14:20:21.316001 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.315983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca9f5f01-a250-4c9c-9d20-6bf062b43632-data-volume\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.326140 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.325941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwl7\" (UniqueName: \"kubernetes.io/projected/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-api-access-8gwl7\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.544643 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.544566 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh"] Sep 10 14:20:21.547744 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:21.547717 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fb0680_1595_43fa_ac3d_c338139b3eb4.slice/crio-24f5f685779d820b8a381ce06419c712cf8f3a5c26d6aaa58a5fde85550ccbf8 WatchSource:0}: Error finding container 24f5f685779d820b8a381ce06419c712cf8f3a5c26d6aaa58a5fde85550ccbf8: Status 404 returned error can't find the container with id 24f5f685779d820b8a381ce06419c712cf8f3a5c26d6aaa58a5fde85550ccbf8 Sep 10 14:20:21.820273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.820164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.820273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.820233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:21.820513 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.820302 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:20:21.820513 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.820355 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:20:21.820513 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.820374 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:22.820358025 +0000 UTC m=+40.674266679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : configmap "kube-rbac-proxy" not found Sep 10 14:20:21.820513 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:21.820419 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:22.820403323 +0000 UTC m=+40.674311999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : secret "insights-runtime-extractor-tls" not found Sep 10 14:20:21.839072 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:21.839036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:22.082025 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.081950 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" event={"ID":"c0fb0680-1595-43fa-ac3d-c338139b3eb4","Type":"ContainerStarted","Data":"24f5f685779d820b8a381ce06419c712cf8f3a5c26d6aaa58a5fde85550ccbf8"} Sep 10 14:20:22.262548 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.262509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:22.311195 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.311158 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:22.538050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.537867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:22.638924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.638886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:22.738750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.738711 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:22.828546 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.828456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:22.828706 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:22.828553 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:22.828706 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:22.828630 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:20:22.828706 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:22.828651 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:20:22.828882 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:22.828713 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:24.828691446 +0000 UTC m=+42.682600115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : configmap "kube-rbac-proxy" not found Sep 10 14:20:22.828882 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:22.828740 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:24.828730814 +0000 UTC m=+42.682639473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : secret "insights-runtime-extractor-tls" not found Sep 10 14:20:23.794338 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:23.794316 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8998p_26d0251c-9cab-48df-bd56-a6724f77f293/dns-node-resolver/0.log" Sep 10 14:20:24.089298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.089219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" event={"ID":"c0fb0680-1595-43fa-ac3d-c338139b3eb4","Type":"ContainerStarted","Data":"a88d25f6c36c7f7dfcf7c9ecee46c8a7fc01ea49276469c867c7652ef35b9caf"} Sep 10 14:20:24.089298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.089274 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" event={"ID":"c0fb0680-1595-43fa-ac3d-c338139b3eb4","Type":"ContainerStarted","Data":"7e7162b2577cb0f534aaed118ff130fcc3bf3c3a1077f7ba3af5d714d2c15f31"} Sep 10 14:20:24.104635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.104586 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m7qjh" podStartSLOduration=1.988940285 podStartE2EDuration="4.104571855s" totalImagesPullingTime="2.11563157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:24.104380777 +0000 UTC m=+41.958289454" watchObservedRunningTime="2026-09-10 14:20:24.104571855 +0000 UTC m=+41.958480531" Sep 10 14:20:24.177092 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.177064 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:24.229068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.229042 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:24.845524 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.845496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.845556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:24.845619 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:24.845650 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.845693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:24.845702 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls podName:649a3eab-a879-4a4b-9d33-5e3d621a5953 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:32.845688609 +0000 UTC m=+50.699597308 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls") pod "dns-default-25qhx" (UID: "649a3eab-a879-4a4b-9d33-5e3d621a5953") : secret "dns-default-metrics-tls" not found Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:24.845728 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:28.845716768 +0000 UTC m=+46.699625425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : configmap "kube-rbac-proxy" not found Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:24.845745 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:20:24.845878 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:24.845769 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:28.845762143 +0000 UTC m=+46.699670796 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : secret "insights-runtime-extractor-tls" not found Sep 10 14:20:24.993366 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:24.993340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pwhpf_a1aa96c3-14fa-4594-ba74-97b26ff67379/node-ca/0.log" Sep 10 14:20:25.380513 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:25.380477 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:25.794209 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:25.794140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gb9tp_5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40/serve-healthcheck-canary/0.log" Sep 10 14:20:25.994684 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:25.994654 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m7qjh_c0fb0680-1595-43fa-ac3d-c338139b3eb4/migrator/0.log" Sep 10 14:20:26.193512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:26.193483 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m7qjh_c0fb0680-1595-43fa-ac3d-c338139b3eb4/graceful-termination/0.log" Sep 10 14:20:28.276730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:28.276688 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:28.774480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:28.774446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:28.875823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:28.875778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:28.875955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:28.875838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:28.875955 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:28.875922 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 14:20:28.875955 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:28.875953 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:20:28.876129 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:28.876003 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:36.875982619 +0000 UTC m=+54.729891297 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : configmap "kube-rbac-proxy" not found Sep 10 14:20:28.876129 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:28.876024 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:36.876014838 +0000 UTC m=+54.729923492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : secret "insights-runtime-extractor-tls" not found Sep 10 14:20:29.134420 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:29.134390 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:29.238077 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:29.238039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:29.337105 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:29.337070 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:29.449419 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:29.449338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:29.551611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:29.551581 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:29.628026 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:29.627996 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:30.137163 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:30.137126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:30.300270 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:30.300217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:30.708730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:30.708699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:31.098239 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:31.098151 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:31.328590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:31.328554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:31.694989 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:31.694956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:32.297346 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.297311 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:32.464449 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.464420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:32.628878 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.628848 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:32.729918 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.729873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:32.894039 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.893956 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:32.905888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.905866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:32.908642 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.908622 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/649a3eab-a879-4a4b-9d33-5e3d621a5953-metrics-tls\") pod \"dns-default-25qhx\" (UID: \"649a3eab-a879-4a4b-9d33-5e3d621a5953\") " pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:32.917628 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:32.917609 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:33.185439 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:33.185368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-25qhx"] Sep 10 14:20:33.188435 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:33.188410 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649a3eab_a879_4a4b_9d33_5e3d621a5953.slice/crio-4008a6d7ac6737f35ce45432ab2eae24420691209e8befe9e7eff75f3d63db67 WatchSource:0}: Error finding container 4008a6d7ac6737f35ce45432ab2eae24420691209e8befe9e7eff75f3d63db67: Status 404 returned error can't find the container with id 4008a6d7ac6737f35ce45432ab2eae24420691209e8befe9e7eff75f3d63db67 Sep 10 14:20:33.295512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:33.295472 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:33.351597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:33.351563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:33.893329 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:33.893297 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:34.115181 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:34.115137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25qhx" event={"ID":"649a3eab-a879-4a4b-9d33-5e3d621a5953","Type":"ContainerStarted","Data":"4008a6d7ac6737f35ce45432ab2eae24420691209e8befe9e7eff75f3d63db67"} Sep 10 14:20:34.116577 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:34.116551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:34.640109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:34.640061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:34.741113 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:34.741075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:34.958042 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:34.957948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:35.095623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:35.095589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:35.294038 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:35.293959 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:35.827346 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:35.827304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:36.122749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.122671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25qhx" event={"ID":"649a3eab-a879-4a4b-9d33-5e3d621a5953","Type":"ContainerStarted","Data":"9db2706fac87beac750e83d3d3aced64bd00a5033cc2650167721028aec20bef"} Sep 10 14:20:36.122749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.122742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-25qhx" event={"ID":"649a3eab-a879-4a4b-9d33-5e3d621a5953","Type":"ContainerStarted","Data":"dc42b2a3e9b763becfbd0c697cadb288215f154bfc309e83be5deba7863533d6"} Sep 10 14:20:36.123012 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.122806 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:36.141247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.141201 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-25qhx" podStartSLOduration=18.013007251 podStartE2EDuration="20.141188653s" totalImagesPullingTime="2.128181402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:36.14051572 +0000 UTC m=+53.994424396" watchObservedRunningTime="2026-09-10 14:20:36.141188653 +0000 UTC m=+53.995097328" Sep 10 14:20:36.324329 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.324300 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:36.427085 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.427050 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:36.694391 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.694301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:36.900300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.900257 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:36.935365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.935338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:36.935487 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.935432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:36.935487 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:36.935476 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 14:20:36.935594 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:20:36.935535 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls podName:ca9f5f01-a250-4c9c-9d20-6bf062b43632 nodeName:}" failed. No retries permitted until 2026-09-10 14:20:52.935520925 +0000 UTC m=+70.789429579 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k5ngq" (UID: "ca9f5f01-a250-4c9c-9d20-6bf062b43632") : secret "insights-runtime-extractor-tls" not found Sep 10 14:20:36.936003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:36.935984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ca9f5f01-a250-4c9c-9d20-6bf062b43632-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:37.297447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:37.297409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:37.698499 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:37.698462 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:38.491356 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:38.491315 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:38.895864 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:38.895823 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:39.092178 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:39.092141 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:39.290623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:39.290541 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:39.494037 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:39.494006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:43.063149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:43.063122 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrjvz" Sep 10 14:20:45.193313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:45.193274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:45.250266 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:45.250236 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:46.127292 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:46.127256 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-25qhx" Sep 10 14:20:47.410663 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.410629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:47.411101 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.410687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:47.413561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.413536 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 14:20:47.414504 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.414489 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 14:20:47.423690 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.423663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7fa850c7-5397-4be3-80a9-2afab7eca821-original-pull-secret\") pod \"global-pull-secret-syncer-9pl2k\" (UID: \"7fa850c7-5397-4be3-80a9-2afab7eca821\") " pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:47.423690 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.423683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16991ff6-9847-4586-b642-2db22ae33842-metrics-certs\") pod \"network-metrics-daemon-q996v\" (UID: \"16991ff6-9847-4586-b642-2db22ae33842\") " pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:47.511746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.511713 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:47.515233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.515216 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 14:20:47.525467 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.525449 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 14:20:47.535364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.535333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2vrr\" (UniqueName: \"kubernetes.io/projected/8f84440f-6fb4-469c-bc1b-ab69e65c87e2-kube-api-access-x2vrr\") pod \"network-check-target-7h85z\" (UID: \"8f84440f-6fb4-469c-bc1b-ab69e65c87e2\") " pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:47.572805 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.572776 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q47dh\"" Sep 10 14:20:47.576242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.576227 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5vvv\"" Sep 10 14:20:47.578323 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.578297 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:47.580485 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.580469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9pl2k" Sep 10 14:20:47.584580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.584558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q996v" Sep 10 14:20:47.859850 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.859818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9pl2k"] Sep 10 14:20:47.863016 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.862990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q996v"] Sep 10 14:20:47.863689 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:47.863663 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa850c7_5397_4be3_80a9_2afab7eca821.slice/crio-78e29bfc0fee2a2c8cd5d4a0d8ea9e9a98cfe2f4d2cbe35daada935e59463406 WatchSource:0}: Error finding container 78e29bfc0fee2a2c8cd5d4a0d8ea9e9a98cfe2f4d2cbe35daada935e59463406: Status 404 returned error can't find the container with id 78e29bfc0fee2a2c8cd5d4a0d8ea9e9a98cfe2f4d2cbe35daada935e59463406 Sep 10 14:20:47.864144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:47.864067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7h85z"] Sep 10 14:20:47.866396 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:47.866372 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f84440f_6fb4_469c_bc1b_ab69e65c87e2.slice/crio-d7ed375e9ae2191f1158a24902fbd4b30b6e344ae93ee412e3fd4e52d6bb3f6d WatchSource:0}: Error finding container d7ed375e9ae2191f1158a24902fbd4b30b6e344ae93ee412e3fd4e52d6bb3f6d: Status 404 returned error can't find the container with id d7ed375e9ae2191f1158a24902fbd4b30b6e344ae93ee412e3fd4e52d6bb3f6d Sep 10 14:20:47.867137 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:47.867114 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16991ff6_9847_4586_b642_2db22ae33842.slice/crio-05d762a4dad13825282f60c035eeff48a0f2ebff91ea2a5b36dd3e919fedd1ce WatchSource:0}: Error finding container 05d762a4dad13825282f60c035eeff48a0f2ebff91ea2a5b36dd3e919fedd1ce: Status 404 returned error can't find the container with id 05d762a4dad13825282f60c035eeff48a0f2ebff91ea2a5b36dd3e919fedd1ce Sep 10 14:20:48.156544 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.156512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7h85z" event={"ID":"8f84440f-6fb4-469c-bc1b-ab69e65c87e2","Type":"ContainerStarted","Data":"d7ed375e9ae2191f1158a24902fbd4b30b6e344ae93ee412e3fd4e52d6bb3f6d"} Sep 10 14:20:48.157630 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.157607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q996v" event={"ID":"16991ff6-9847-4586-b642-2db22ae33842","Type":"ContainerStarted","Data":"05d762a4dad13825282f60c035eeff48a0f2ebff91ea2a5b36dd3e919fedd1ce"} Sep 10 14:20:48.158589 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.158573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9pl2k" event={"ID":"7fa850c7-5397-4be3-80a9-2afab7eca821","Type":"ContainerStarted","Data":"78e29bfc0fee2a2c8cd5d4a0d8ea9e9a98cfe2f4d2cbe35daada935e59463406"} Sep 10 14:20:48.451983 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.451908 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p"] Sep 10 14:20:48.454138 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.454114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.457650 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.457470 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 14:20:48.458670 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.458648 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 14:20:48.459847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.459067 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 14:20:48.459847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.459277 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 14:20:48.459847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.459465 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 14:20:48.459847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.459680 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 14:20:48.462655 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.462613 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 14:20:48.482811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.482626 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p"] Sep 10 14:20:48.517615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.517373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skfbz\" (UniqueName: \"kubernetes.io/projected/2d39b5a4-00c6-4c33-8acb-534349efe479-kube-api-access-skfbz\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.517615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.517423 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-hub\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.517615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.517451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.517615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.517478 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d39b5a4-00c6-4c33-8acb-534349efe479-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.517615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.517503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.517615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.517531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-ca\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.535681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.535641 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-655f9c8b9f-9vgf4"] Sep 10 14:20:48.538538 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.538024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.543270 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.541698 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cgxdb\"" Sep 10 14:20:48.543270 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.541972 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 14:20:48.543270 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.542189 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 14:20:48.543270 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.543103 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 14:20:48.544763 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.544694 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:48.548106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.548035 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 14:20:48.554314 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.554256 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655f9c8b9f-9vgf4"] Sep 10 14:20:48.618035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-bound-sa-token\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618218 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/684530a2-e166-41a2-8f5c-728e8eaad971-trusted-ca\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618218 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-registry-tls\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618218 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/684530a2-e166-41a2-8f5c-728e8eaad971-ca-trust-extracted\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618218 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/684530a2-e166-41a2-8f5c-728e8eaad971-installation-pull-secrets\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618218 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdjzh\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-kube-api-access-xdjzh\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skfbz\" (UniqueName: \"kubernetes.io/projected/2d39b5a4-00c6-4c33-8acb-534349efe479-kube-api-access-skfbz\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.618480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/684530a2-e166-41a2-8f5c-728e8eaad971-image-registry-private-configuration\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.618480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-hub\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.618480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618375 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.618691 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d39b5a4-00c6-4c33-8acb-534349efe479-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.618691 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618684 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.618818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618711 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-ca\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.618818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.618738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/684530a2-e166-41a2-8f5c-728e8eaad971-registry-certificates\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.619442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.619419 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d39b5a4-00c6-4c33-8acb-534349efe479-additional-service-ca\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.624189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.624163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.626260 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.626235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-hub\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.629059 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.629017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skfbz\" (UniqueName: \"kubernetes.io/projected/2d39b5a4-00c6-4c33-8acb-534349efe479-kube-api-access-skfbz\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.632890 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.632621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.635486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.634933 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2d39b5a4-00c6-4c33-8acb-534349efe479-ca\") pod \"cluster-proxy-proxy-agent-57c75f8f-25w4p\" (UID: \"2d39b5a4-00c6-4c33-8acb-534349efe479\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.642532 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.642457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.719903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/684530a2-e166-41a2-8f5c-728e8eaad971-trusted-ca\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.719972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-registry-tls\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720011 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/684530a2-e166-41a2-8f5c-728e8eaad971-ca-trust-extracted\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/684530a2-e166-41a2-8f5c-728e8eaad971-installation-pull-secrets\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdjzh\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-kube-api-access-xdjzh\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/684530a2-e166-41a2-8f5c-728e8eaad971-image-registry-private-configuration\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/684530a2-e166-41a2-8f5c-728e8eaad971-registry-certificates\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720326 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-bound-sa-token\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.720750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720513 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/684530a2-e166-41a2-8f5c-728e8eaad971-ca-trust-extracted\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.721333 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.720985 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/684530a2-e166-41a2-8f5c-728e8eaad971-trusted-ca\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.721333 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.721248 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/684530a2-e166-41a2-8f5c-728e8eaad971-registry-certificates\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.723538 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.723469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/684530a2-e166-41a2-8f5c-728e8eaad971-image-registry-private-configuration\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.723898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.723873 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/684530a2-e166-41a2-8f5c-728e8eaad971-installation-pull-secrets\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.725672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.725649 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-registry-tls\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.737258 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.737233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdjzh\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-kube-api-access-xdjzh\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.737425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.737400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:48.739390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.739350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/684530a2-e166-41a2-8f5c-728e8eaad971-bound-sa-token\") pod \"image-registry-655f9c8b9f-9vgf4\" (UID: \"684530a2-e166-41a2-8f5c-728e8eaad971\") " pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.777989 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.777956 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" Sep 10 14:20:48.836161 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.836013 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:48.853120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.852952 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:48.954521 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:48.954432 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:49.052056 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:49.051951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:49.070881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:49.070819 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p"] Sep 10 14:20:49.109005 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:49.108964 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d39b5a4_00c6_4c33_8acb_534349efe479.slice/crio-5627a6483b6fb27811ad394e75cd27dd68a11d67d19a5ab2df4cf8fe3e2e3e53 WatchSource:0}: Error finding container 5627a6483b6fb27811ad394e75cd27dd68a11d67d19a5ab2df4cf8fe3e2e3e53: Status 404 returned error can't find the container with id 5627a6483b6fb27811ad394e75cd27dd68a11d67d19a5ab2df4cf8fe3e2e3e53 Sep 10 14:20:49.120984 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:49.120920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:49.169040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:49.168961 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" event={"ID":"2d39b5a4-00c6-4c33-8acb-534349efe479","Type":"ContainerStarted","Data":"5627a6483b6fb27811ad394e75cd27dd68a11d67d19a5ab2df4cf8fe3e2e3e53"} Sep 10 14:20:49.390882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:49.390853 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655f9c8b9f-9vgf4"] Sep 10 14:20:49.394422 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:49.394390 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod684530a2_e166_41a2_8f5c_728e8eaad971.slice/crio-4c813afe858c56491b5067494d5a4364e9a79ae511fdd001141a5fc40e457975 WatchSource:0}: Error finding container 4c813afe858c56491b5067494d5a4364e9a79ae511fdd001141a5fc40e457975: Status 404 returned error can't find the container with id 4c813afe858c56491b5067494d5a4364e9a79ae511fdd001141a5fc40e457975 Sep 10 14:20:50.177233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.177192 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" event={"ID":"684530a2-e166-41a2-8f5c-728e8eaad971","Type":"ContainerStarted","Data":"8251c2d320590a92a55befcfe5cf24ff33ddaa22b424e48daecc52ee91fe4f37"} Sep 10 14:20:50.177711 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.177262 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" event={"ID":"684530a2-e166-41a2-8f5c-728e8eaad971","Type":"ContainerStarted","Data":"4c813afe858c56491b5067494d5a4364e9a79ae511fdd001141a5fc40e457975"} Sep 10 14:20:50.178032 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.178011 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:20:50.183334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.183256 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q996v" event={"ID":"16991ff6-9847-4586-b642-2db22ae33842","Type":"ContainerStarted","Data":"68089ca07ef60d4b5a5308ef22375bce49e009f4814e9f87648293f8010f7c56"} Sep 10 14:20:50.183334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.183301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q996v" event={"ID":"16991ff6-9847-4586-b642-2db22ae33842","Type":"ContainerStarted","Data":"63330ea67b7f5b8f5c339c288a13fdfd8d113a0cee38529db00de0808d34385e"} Sep 10 14:20:50.199638 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.199582 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" podStartSLOduration=2.199564861 podStartE2EDuration="2.199564861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:50.198709781 +0000 UTC m=+68.052618481" watchObservedRunningTime="2026-09-10 14:20:50.199564861 +0000 UTC m=+68.053473532" Sep 10 14:20:50.218156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.218001 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q996v" podStartSLOduration=65.949170953 podStartE2EDuration="1m7.217990851s" totalImagesPullingTime="1.268819898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:50.216500659 +0000 UTC m=+68.070409329" watchObservedRunningTime="2026-09-10 14:20:50.217990851 +0000 UTC m=+68.071899527" Sep 10 14:20:50.319407 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.319229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:50.417535 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:50.417431 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:51.803581 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:51.803455 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:51.893333 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:51.893298 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:52.774229 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:52.774186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:52.872946 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:52.872904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:52.957301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:52.957264 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:52.959804 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:52.959768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ca9f5f01-a250-4c9c-9d20-6bf062b43632-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k5ngq\" (UID: \"ca9f5f01-a250-4c9c-9d20-6bf062b43632\") " pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:52.973374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:52.973338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:53.159486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:53.159454 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mdxkp\"" Sep 10 14:20:53.167742 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:53.167719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k5ngq" Sep 10 14:20:55.389077 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:55.389029 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k5ngq"] Sep 10 14:20:55.391636 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:20:55.391610 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9f5f01_a250_4c9c_9d20_6bf062b43632.slice/crio-6398405d63b2943f14e9a2b1385e0fb0f87489efd68c5a6d0214bdfb130ba3ce WatchSource:0}: Error finding container 6398405d63b2943f14e9a2b1385e0fb0f87489efd68c5a6d0214bdfb130ba3ce: Status 404 returned error can't find the container with id 6398405d63b2943f14e9a2b1385e0fb0f87489efd68c5a6d0214bdfb130ba3ce Sep 10 14:20:56.205622 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.205583 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9pl2k" event={"ID":"7fa850c7-5397-4be3-80a9-2afab7eca821","Type":"ContainerStarted","Data":"bd9b1fa69d9e9a91d7a79af4188706e0193ebe0c8bf7c78ebbcd317a9c2b3e54"} Sep 10 14:20:56.207651 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.207627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" event={"ID":"2d39b5a4-00c6-4c33-8acb-534349efe479","Type":"ContainerStarted","Data":"8ac1c760f5ae6f17f61c1de00859d4342b5cf956ff88232b19303460785265ac"} Sep 10 14:20:56.207787 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.207662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" event={"ID":"2d39b5a4-00c6-4c33-8acb-534349efe479","Type":"ContainerStarted","Data":"2ee292c81262b8047b7d505083d056f912ba62e779be8b7948745f11603f91b5"} Sep 10 14:20:56.207787 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.207681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" event={"ID":"2d39b5a4-00c6-4c33-8acb-534349efe479","Type":"ContainerStarted","Data":"20309c9f823c5d357b8e5121a879c66c55d37536fc51acd516e9e9da587793c1"} Sep 10 14:20:56.209392 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.209365 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5ngq" event={"ID":"ca9f5f01-a250-4c9c-9d20-6bf062b43632","Type":"ContainerStarted","Data":"cb04d11dd030392fe9bcb3a7a59e5b33ec8063f20771d30d9e0136e8df27dcef"} Sep 10 14:20:56.209479 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.209415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5ngq" event={"ID":"ca9f5f01-a250-4c9c-9d20-6bf062b43632","Type":"ContainerStarted","Data":"38a10725e8482a67a3cf1aea9d0cb52b7a1f1d556465f2880f6a85f04025b284"} Sep 10 14:20:56.209479 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.209443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5ngq" event={"ID":"ca9f5f01-a250-4c9c-9d20-6bf062b43632","Type":"ContainerStarted","Data":"6398405d63b2943f14e9a2b1385e0fb0f87489efd68c5a6d0214bdfb130ba3ce"} Sep 10 14:20:56.210742 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.210687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7h85z" event={"ID":"8f84440f-6fb4-469c-bc1b-ab69e65c87e2","Type":"ContainerStarted","Data":"fd2fd582e5755c99d94b29995a57b17aa1b4d00b7eda7c596e07161eaddba891"} Sep 10 14:20:56.210838 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.210808 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:20:56.227017 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.223744 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9pl2k" podStartSLOduration=65.96998594 podStartE2EDuration="1m13.223730359s" totalImagesPullingTime="7.253744419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:56.223366088 +0000 UTC m=+74.077274765" watchObservedRunningTime="2026-09-10 14:20:56.223730359 +0000 UTC m=+74.077639037" Sep 10 14:20:56.260039 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.260011 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:56.265118 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.265080 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7h85z" podStartSLOduration=66.014084356 podStartE2EDuration="1m13.265067653s" totalImagesPullingTime="7.250983297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:56.245087282 +0000 UTC m=+74.098995960" watchObservedRunningTime="2026-09-10 14:20:56.265067653 +0000 UTC m=+74.118976329" Sep 10 14:20:56.265307 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.265285 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-57c75f8f-25w4p" podStartSLOduration=2.204117598 podStartE2EDuration="8.265279144s" totalImagesPullingTime="6.061161546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:20:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:56.264877679 +0000 UTC m=+74.118786354" watchObservedRunningTime="2026-09-10 14:20:56.265279144 +0000 UTC m=+74.119187820" Sep 10 14:20:56.362584 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.362547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:56.592996 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.592886 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:56.697774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:56.697739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:58.220253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:58.220218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k5ngq" event={"ID":"ca9f5f01-a250-4c9c-9d20-6bf062b43632","Type":"ContainerStarted","Data":"753d0df3d2b0cd2af3a33ab05a93442d1f1cce4002f938384f795effc66d9e3e"} Sep 10 14:20:58.249577 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:58.249529 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k5ngq" podStartSLOduration=35.488449912 podStartE2EDuration="37.249511374s" totalImagesPullingTime="1.761061462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:20:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:20:58.248389466 +0000 UTC m=+76.102298132" watchObservedRunningTime="2026-09-10 14:20:58.249511374 +0000 UTC m=+76.103420044" Sep 10 14:20:59.020077 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:59.020032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:59.794244 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:59.794207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:20:59.892759 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:20:59.892709 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:00.021280 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:00.021244 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:00.178840 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:00.178804 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:00.279340 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:00.279307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:00.403220 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:00.403181 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:00.634496 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:00.634469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:01.597056 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:01.597012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:01.753630 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:01.753592 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:01.850995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:01.850911 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.087686 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.087651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.221864 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.221824 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.473110 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.473027 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sm8fz"] Sep 10 14:21:02.475914 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.475897 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.479179 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.479157 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 14:21:02.479283 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.479205 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 14:21:02.480597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.480580 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7t8jk\"" Sep 10 14:21:02.480685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.480621 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 14:21:02.480733 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.480690 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 14:21:02.480776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.480630 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 14:21:02.480849 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.480621 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 14:21:02.511906 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.511871 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.584103 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.584074 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.636194 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636160 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-wtmp\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-tls\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-metrics-client-ca\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9h5\" (UniqueName: \"kubernetes.io/projected/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-kube-api-access-rn9h5\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-sys\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-root\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.636624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.636472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-textfile\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.652955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.652918 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.736988 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.736904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-sys\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.736988 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.736939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-root\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.736988 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.736956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-textfile\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.736995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-wtmp\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-sys\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-tls\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-root\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-metrics-client-ca\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737163 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-wtmp\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737221 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9h5\" (UniqueName: \"kubernetes.io/projected/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-kube-api-access-rn9h5\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737528 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-textfile\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737639 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-metrics-client-ca\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.737700 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.737677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-accelerators-collector-config\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.739298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.739275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-tls\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.739403 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.739327 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.750672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.750650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9h5\" (UniqueName: \"kubernetes.io/projected/f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd-kube-api-access-rn9h5\") pod \"node-exporter-sm8fz\" (UID: \"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd\") " pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.755424 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.755391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.785341 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.785311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sm8fz" Sep 10 14:21:02.875733 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.875695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:02.977188 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:02.977145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:03.237205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:03.237165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm8fz" event={"ID":"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd","Type":"ContainerStarted","Data":"ee5542209cc9e3d599c9012d0b54e6ba35b43f0742230967f60cd6cceeac388a"} Sep 10 14:21:03.237471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:03.237449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:03.343195 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:03.343164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:03.550699 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:03.550600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:04.242348 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:04.242315 2576 generic.go:356] "Generic (PLEG): container finished" podID="f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd" containerID="f1c31025b8f85f698e2120829931343d9aa071b180c05346f06fde2f41718e02" exitCode=0 Sep 10 14:21:04.242665 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:04.242366 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm8fz" event={"ID":"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd","Type":"ContainerDied","Data":"f1c31025b8f85f698e2120829931343d9aa071b180c05346f06fde2f41718e02"} Sep 10 14:21:05.247661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.247544 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm8fz" event={"ID":"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd","Type":"ContainerStarted","Data":"22545981bca3d717647a76e5937bcd9319f19f1383eb6aee347c9c9033a2b3ae"} Sep 10 14:21:05.247661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.247594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sm8fz" event={"ID":"f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd","Type":"ContainerStarted","Data":"bda0e7ff47e42d5d16a0218de8ab84f572fd00ad2ee74f517e166fa0e2e33d18"} Sep 10 14:21:05.289602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.289548 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sm8fz" podStartSLOduration=1.9342124040000002 podStartE2EDuration="3.289532904s" totalImagesPullingTime="1.292775368s" totalInitContainerRuntime="62.545132ms" isStatefulPod=true podCreationTimestamp="2026-09-10 14:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:05.28908304 +0000 UTC m=+83.142991732" watchObservedRunningTime="2026-09-10 14:21:05.289532904 +0000 UTC m=+83.143441579" Sep 10 14:21:05.427740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.427712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:05.498392 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.498312 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:05.599129 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.599087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:05.701567 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:05.701508 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:06.807954 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.807925 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-688cf66c9d-v529b"] Sep 10 14:21:06.810318 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.810304 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.813556 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.813532 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 14:21:06.814934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.814909 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 14:21:06.814934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.814936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9mqv2bep6grg1\"" Sep 10 14:21:06.815116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.814953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 14:21:06.815116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.814991 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 14:21:06.815116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.814936 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2ftl6\"" Sep 10 14:21:06.826422 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.826399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-688cf66c9d-v529b"] Sep 10 14:21:06.968746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.968706 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-secret-metrics-server-tls\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.968938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.968766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b00406b1-e4ce-4a29-99d1-01fbaed628ad-audit-log\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.968938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.968830 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b00406b1-e4ce-4a29-99d1-01fbaed628ad-metrics-server-audit-profiles\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.968938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.968912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpn99\" (UniqueName: \"kubernetes.io/projected/b00406b1-e4ce-4a29-99d1-01fbaed628ad-kube-api-access-xpn99\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.969045 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.968939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-client-ca-bundle\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.969045 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.969001 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-secret-metrics-server-client-certs\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:06.969045 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:06.969033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b00406b1-e4ce-4a29-99d1-01fbaed628ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.002616 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.002578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:07.069665 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b00406b1-e4ce-4a29-99d1-01fbaed628ad-audit-log\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.069665 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069608 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b00406b1-e4ce-4a29-99d1-01fbaed628ad-metrics-server-audit-profiles\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.069665 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpn99\" (UniqueName: \"kubernetes.io/projected/b00406b1-e4ce-4a29-99d1-01fbaed628ad-kube-api-access-xpn99\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.069665 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-client-ca-bundle\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.070062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-secret-metrics-server-client-certs\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.070062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b00406b1-e4ce-4a29-99d1-01fbaed628ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.070062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.069857 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-secret-metrics-server-tls\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.070062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.070044 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/b00406b1-e4ce-4a29-99d1-01fbaed628ad-audit-log\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.070531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.070506 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b00406b1-e4ce-4a29-99d1-01fbaed628ad-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.070703 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.070639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/b00406b1-e4ce-4a29-99d1-01fbaed628ad-metrics-server-audit-profiles\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.072201 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.072176 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-client-ca-bundle\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.072334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.072315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-secret-metrics-server-tls\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.072408 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.072334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/b00406b1-e4ce-4a29-99d1-01fbaed628ad-secret-metrics-server-client-certs\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.078399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.078380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpn99\" (UniqueName: \"kubernetes.io/projected/b00406b1-e4ce-4a29-99d1-01fbaed628ad-kube-api-access-xpn99\") pod \"metrics-server-688cf66c9d-v529b\" (UID: \"b00406b1-e4ce-4a29-99d1-01fbaed628ad\") " pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.083289 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.083263 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:07.119385 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.119357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:07.388672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.388638 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-688cf66c9d-v529b"] Sep 10 14:21:07.392138 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:21:07.392113 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00406b1_e4ce_4a29_99d1_01fbaed628ad.slice/crio-0a8d4a0de1113fd514ea2c587dec94163559631ea7c694d3f0ba0a37a9c9c0e4 WatchSource:0}: Error finding container 0a8d4a0de1113fd514ea2c587dec94163559631ea7c694d3f0ba0a37a9c9c0e4: Status 404 returned error can't find the container with id 0a8d4a0de1113fd514ea2c587dec94163559631ea7c694d3f0ba0a37a9c9c0e4 Sep 10 14:21:07.470028 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:07.469989 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:08.260955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:08.260912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" event={"ID":"b00406b1-e4ce-4a29-99d1-01fbaed628ad","Type":"ContainerStarted","Data":"0a8d4a0de1113fd514ea2c587dec94163559631ea7c694d3f0ba0a37a9c9c0e4"} Sep 10 14:21:08.774438 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:08.774386 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:09.266299 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:09.266266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" event={"ID":"b00406b1-e4ce-4a29-99d1-01fbaed628ad","Type":"ContainerStarted","Data":"09510943f75077f725f63154ea522c3a73757ca8a330fe665b5841c4ab80be3e"} Sep 10 14:21:09.293859 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:09.293764 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" podStartSLOduration=1.7529480629999998 podStartE2EDuration="3.293745054s" totalImagesPullingTime="1.540796991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:09.292687004 +0000 UTC m=+87.146595677" watchObservedRunningTime="2026-09-10 14:21:09.293745054 +0000 UTC m=+87.147653723" Sep 10 14:21:11.428056 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.428025 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75b97bd97c-bvpfn"] Sep 10 14:21:11.429924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.429909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.434124 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434099 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qcks9\"" Sep 10 14:21:11.434396 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434376 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 14:21:11.434396 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434390 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 14:21:11.434598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434416 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 14:21:11.434598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434380 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 14:21:11.434598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 14:21:11.434598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434382 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 14:21:11.434598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.434535 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 14:21:11.440772 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.440752 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b97bd97c-bvpfn"] Sep 10 14:21:11.604519 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.604477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-serving-cert\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.604519 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.604527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-console-config\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.604776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.604545 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-oauth-config\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.604776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.604646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-oauth-serving-cert\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.604776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.604696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-service-ca\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.604776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.604730 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvq8c\" (UniqueName: \"kubernetes.io/projected/c5eeac31-8658-437c-b84f-706bfc9c822e-kube-api-access-fvq8c\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.705953 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.705852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-service-ca\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.705953 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.705905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvq8c\" (UniqueName: \"kubernetes.io/projected/c5eeac31-8658-437c-b84f-706bfc9c822e-kube-api-access-fvq8c\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.705953 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.705950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-serving-cert\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.706207 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.705981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-console-config\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.706207 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.706001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-oauth-config\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.706207 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.706045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-oauth-serving-cert\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.706661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.706634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-console-config\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.706764 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.706631 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-service-ca\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.706764 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.706727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-oauth-serving-cert\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.708347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.708326 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-serving-cert\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.708347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.708336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-oauth-config\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.716886 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.716866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvq8c\" (UniqueName: \"kubernetes.io/projected/c5eeac31-8658-437c-b84f-706bfc9c822e-kube-api-access-fvq8c\") pod \"console-75b97bd97c-bvpfn\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.738356 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.738331 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:11.755041 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:11.755018 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:12.007433 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.007409 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75b97bd97c-bvpfn"] Sep 10 14:21:12.009733 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:21:12.009711 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5eeac31_8658_437c_b84f_706bfc9c822e.slice/crio-d58683aee8fa3b292bfc110a88955fcbf3d276939dd9293c7e8efe5feebaa5a2 WatchSource:0}: Error finding container d58683aee8fa3b292bfc110a88955fcbf3d276939dd9293c7e8efe5feebaa5a2: Status 404 returned error can't find the container with id d58683aee8fa3b292bfc110a88955fcbf3d276939dd9293c7e8efe5feebaa5a2 Sep 10 14:21:12.194520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.194491 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-655f9c8b9f-9vgf4" Sep 10 14:21:12.276592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.276517 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b97bd97c-bvpfn" event={"ID":"c5eeac31-8658-437c-b84f-706bfc9c822e","Type":"ContainerStarted","Data":"d58683aee8fa3b292bfc110a88955fcbf3d276939dd9293c7e8efe5feebaa5a2"} Sep 10 14:21:12.332868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.332832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:12.390882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.390845 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:12.449052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.449019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:12.601785 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.601710 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:12.663144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:12.663104 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:15.270116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:15.270033 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:15.290040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:15.290012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b97bd97c-bvpfn" event={"ID":"c5eeac31-8658-437c-b84f-706bfc9c822e","Type":"ContainerStarted","Data":"aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0"} Sep 10 14:21:15.313107 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:15.313061 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75b97bd97c-bvpfn" podStartSLOduration=1.408034776 podStartE2EDuration="4.313047972s" totalImagesPullingTime="2.905013196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:21:15.311124678 +0000 UTC m=+93.165033354" watchObservedRunningTime="2026-09-10 14:21:15.313047972 +0000 UTC m=+93.166956647" Sep 10 14:21:15.360911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:15.360881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:18.139502 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:18.139465 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:18.286466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:18.286427 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:20.775427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:20.775398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:20.876504 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:20.876458 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:21.738583 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:21.738551 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:21.738760 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:21.738606 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:21.743549 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:21.743526 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:22.318446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:22.318418 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:26.705181 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:26.705146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:26.799586 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:26.799551 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:27.119831 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:27.119730 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:27.119831 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:27.119812 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:27.217252 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:27.217224 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7h85z" Sep 10 14:21:28.404274 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:28.404234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:28.505537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:28.505505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:30.192365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:30.192338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:30.292097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:30.292057 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:31.853190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:31.853152 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:31.914233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:31.914197 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:32.871468 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:32.871432 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75b97bd97c-bvpfn"] Sep 10 14:21:34.007716 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:34.007681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:34.108735 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:34.108700 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:34.830626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:34.830588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:34.930949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:34.930906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:41.144620 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:41.144592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gb9tp_5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40/serve-healthcheck-canary/0.log" Sep 10 14:21:45.198468 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:45.198417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:45.299165 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:45.299133 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:47.124883 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:47.124852 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:47.128672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:47.128651 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-688cf66c9d-v529b" Sep 10 14:21:53.860630 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:53.860591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:53.960634 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:53.960603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:21:57.890881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:57.890837 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75b97bd97c-bvpfn" podUID="c5eeac31-8658-437c-b84f-706bfc9c822e" containerName="console" containerID="cri-o://aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0" gracePeriod=15 Sep 10 14:21:58.121578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.121554 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:58.143027 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.142954 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvq8c\" (UniqueName: \"kubernetes.io/projected/c5eeac31-8658-437c-b84f-706bfc9c822e-kube-api-access-fvq8c\") pod \"c5eeac31-8658-437c-b84f-706bfc9c822e\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " Sep 10 14:21:58.143027 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.142988 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-serving-cert\") pod \"c5eeac31-8658-437c-b84f-706bfc9c822e\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " Sep 10 14:21:58.143210 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143040 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-service-ca\") pod \"c5eeac31-8658-437c-b84f-706bfc9c822e\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " Sep 10 14:21:58.143210 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143070 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-console-config\") pod \"c5eeac31-8658-437c-b84f-706bfc9c822e\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " Sep 10 14:21:58.143210 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143105 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-oauth-config\") pod \"c5eeac31-8658-437c-b84f-706bfc9c822e\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " Sep 10 14:21:58.143355 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143211 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-oauth-serving-cert\") pod \"c5eeac31-8658-437c-b84f-706bfc9c822e\" (UID: \"c5eeac31-8658-437c-b84f-706bfc9c822e\") " Sep 10 14:21:58.143749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-service-ca" (OuterVolumeSpecName: "service-ca") pod "c5eeac31-8658-437c-b84f-706bfc9c822e" (UID: "c5eeac31-8658-437c-b84f-706bfc9c822e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:21:58.143749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143465 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-console-config" (OuterVolumeSpecName: "console-config") pod "c5eeac31-8658-437c-b84f-706bfc9c822e" (UID: "c5eeac31-8658-437c-b84f-706bfc9c822e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:21:58.143749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.143683 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c5eeac31-8658-437c-b84f-706bfc9c822e" (UID: "c5eeac31-8658-437c-b84f-706bfc9c822e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:21:58.145378 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.145348 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5eeac31-8658-437c-b84f-706bfc9c822e-kube-api-access-fvq8c" (OuterVolumeSpecName: "kube-api-access-fvq8c") pod "c5eeac31-8658-437c-b84f-706bfc9c822e" (UID: "c5eeac31-8658-437c-b84f-706bfc9c822e"). InnerVolumeSpecName "kube-api-access-fvq8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:21:58.145701 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.145680 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c5eeac31-8658-437c-b84f-706bfc9c822e" (UID: "c5eeac31-8658-437c-b84f-706bfc9c822e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:21:58.145774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.145701 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c5eeac31-8658-437c-b84f-706bfc9c822e" (UID: "c5eeac31-8658-437c-b84f-706bfc9c822e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:21:58.243948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.243907 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:21:58.243948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.243940 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-console-config\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:21:58.243948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.243948 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-oauth-config\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:21:58.243948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.243957 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5eeac31-8658-437c-b84f-706bfc9c822e-oauth-serving-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:21:58.244190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.243968 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvq8c\" (UniqueName: \"kubernetes.io/projected/c5eeac31-8658-437c-b84f-706bfc9c822e-kube-api-access-fvq8c\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:21:58.244190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.243977 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5eeac31-8658-437c-b84f-706bfc9c822e-console-serving-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:21:58.432842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.432734 2576 generic.go:356] "Generic (PLEG): container finished" podID="c5eeac31-8658-437c-b84f-706bfc9c822e" containerID="aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0" exitCode=0 Sep 10 14:21:58.432842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.432786 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75b97bd97c-bvpfn" Sep 10 14:21:58.432842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.432828 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b97bd97c-bvpfn" event={"ID":"c5eeac31-8658-437c-b84f-706bfc9c822e","Type":"ContainerDied","Data":"aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0"} Sep 10 14:21:58.433052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.432881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75b97bd97c-bvpfn" event={"ID":"c5eeac31-8658-437c-b84f-706bfc9c822e","Type":"ContainerDied","Data":"d58683aee8fa3b292bfc110a88955fcbf3d276939dd9293c7e8efe5feebaa5a2"} Sep 10 14:21:58.433052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.432907 2576 scope.go:117] "RemoveContainer" containerID="aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0" Sep 10 14:21:58.445384 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.445359 2576 scope.go:117] "RemoveContainer" containerID="aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0" Sep 10 14:21:58.445653 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:21:58.445631 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0\": container with ID starting with aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0 not found: ID does not exist" containerID="aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0" Sep 10 14:21:58.445705 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.445661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0"} err="failed to get container status \"aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0\": rpc error: code = NotFound desc = could not find container \"aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0\": container with ID starting with aa88752150ce536e99a6a9f3accd0ddea984ff072dafd925f9551ef6e1e485d0 not found: ID does not exist" Sep 10 14:21:58.456335 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.456311 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75b97bd97c-bvpfn"] Sep 10 14:21:58.461643 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.461623 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75b97bd97c-bvpfn"] Sep 10 14:21:58.758947 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:21:58.758871 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5eeac31-8658-437c-b84f-706bfc9c822e" path="/var/lib/kubelet/pods/c5eeac31-8658-437c-b84f-706bfc9c822e/volumes" Sep 10 14:22:18.582484 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:18.582441 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:18.638950 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:18.638909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:30.433153 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:30.433113 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:30.534445 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:30.534410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:45.212470 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:45.212415 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:45.313510 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:45.313480 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:45.415059 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:45.415025 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:46.278093 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.278063 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bd74ccb95-xzskc"] Sep 10 14:22:46.278453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.278303 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5eeac31-8658-437c-b84f-706bfc9c822e" containerName="console" Sep 10 14:22:46.278453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.278313 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5eeac31-8658-437c-b84f-706bfc9c822e" containerName="console" Sep 10 14:22:46.278453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.278362 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5eeac31-8658-437c-b84f-706bfc9c822e" containerName="console" Sep 10 14:22:46.282202 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.282186 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.285167 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285145 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qcks9\"" Sep 10 14:22:46.285459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285438 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 10 14:22:46.285596 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285580 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 10 14:22:46.285657 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285627 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 10 14:22:46.285780 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285759 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 10 14:22:46.285874 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285780 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 10 14:22:46.285874 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.285782 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 10 14:22:46.288537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.288422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 10 14:22:46.293730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.293712 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 10 14:22:46.297321 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.297298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd74ccb95-xzskc"] Sep 10 14:22:46.341711 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.341663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:46.381552 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-oauth-serving-cert\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.381688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381562 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-serving-cert\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.381688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381616 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7brn2\" (UniqueName: \"kubernetes.io/projected/997f1b20-c8e5-4679-9e70-b5ee59e1771f-kube-api-access-7brn2\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.381688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-oauth-config\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.381894 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-config\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.381894 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-service-ca\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.381894 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.381778 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-trusted-ca-bundle\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.472505 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.472474 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:46.482279 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7brn2\" (UniqueName: \"kubernetes.io/projected/997f1b20-c8e5-4679-9e70-b5ee59e1771f-kube-api-access-7brn2\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.482353 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-oauth-config\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.482353 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-config\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.482353 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-service-ca\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.482353 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-trusted-ca-bundle\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.482529 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482382 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-oauth-serving-cert\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.482529 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.482418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-serving-cert\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.483109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.483087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-config\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.483221 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.483125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-service-ca\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.483298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.483272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-oauth-serving-cert\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.483682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.483659 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-trusted-ca-bundle\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.484862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.484843 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-oauth-config\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.484936 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.484872 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-serving-cert\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.489934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.489914 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7brn2\" (UniqueName: \"kubernetes.io/projected/997f1b20-c8e5-4679-9e70-b5ee59e1771f-kube-api-access-7brn2\") pod \"console-6bd74ccb95-xzskc\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.572824 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.572734 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:46.594214 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.594187 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:46.864757 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:46.864673 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bd74ccb95-xzskc"] Sep 10 14:22:46.867680 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:22:46.867653 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997f1b20_c8e5_4679_9e70_b5ee59e1771f.slice/crio-f339bea517b9703dcc6fe8fb51a1065668b4709b6c80dddfb64b9fa7ce90c8cb WatchSource:0}: Error finding container f339bea517b9703dcc6fe8fb51a1065668b4709b6c80dddfb64b9fa7ce90c8cb: Status 404 returned error can't find the container with id f339bea517b9703dcc6fe8fb51a1065668b4709b6c80dddfb64b9fa7ce90c8cb Sep 10 14:22:47.097480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:47.097440 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:47.165991 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:47.165948 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:47.600442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:47.600353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd74ccb95-xzskc" event={"ID":"997f1b20-c8e5-4679-9e70-b5ee59e1771f","Type":"ContainerStarted","Data":"fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59"} Sep 10 14:22:47.600442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:47.600416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd74ccb95-xzskc" event={"ID":"997f1b20-c8e5-4679-9e70-b5ee59e1771f","Type":"ContainerStarted","Data":"f339bea517b9703dcc6fe8fb51a1065668b4709b6c80dddfb64b9fa7ce90c8cb"} Sep 10 14:22:47.618754 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:47.618705 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bd74ccb95-xzskc" podStartSLOduration=1.618691522 podStartE2EDuration="1.618691522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:22:47.616698035 +0000 UTC m=+185.470606711" watchObservedRunningTime="2026-09-10 14:22:47.618691522 +0000 UTC m=+185.472600198" Sep 10 14:22:56.594459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.594412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:56.594925 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.594481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:56.599384 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.599364 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:56.634053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.634031 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:22:56.777300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.777247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:56.877531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.877496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:22:56.929592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:22:56.929560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:11.691513 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:11.691475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:11.749900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:11.749864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:17.912407 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:17.912371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:18.005365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:18.005337 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:26.871423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:26.871381 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:26.923995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:26.923970 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:44.842340 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:44.842304 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:46.196977 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:46.196940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:48.649950 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:48.649914 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:48.710807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:48.710731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:49.829347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:49.829292 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:49.946311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:49.946271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:50.109663 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:50.109555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:52.383621 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:52.383579 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:52.583441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:52.583414 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:55.455702 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:55.455671 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:55.521217 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:55.521178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:56.562204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.562172 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mlcnh"] Sep 10 14:23:56.565208 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.565191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.567861 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.567840 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 14:23:56.569036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.569018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 14:23:56.569036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.569030 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6m54x\"" Sep 10 14:23:56.574978 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.574958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mlcnh"] Sep 10 14:23:56.664716 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.664682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbd100d1-825e-4ff4-abeb-7f4ae2a444aa-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mlcnh\" (UID: \"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.664716 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.664716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7zg\" (UniqueName: \"kubernetes.io/projected/dbd100d1-825e-4ff4-abeb-7f4ae2a444aa-kube-api-access-bp7zg\") pod \"cert-manager-webhook-6486ffd997-mlcnh\" (UID: \"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.765295 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.765270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbd100d1-825e-4ff4-abeb-7f4ae2a444aa-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mlcnh\" (UID: \"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.765418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.765300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7zg\" (UniqueName: \"kubernetes.io/projected/dbd100d1-825e-4ff4-abeb-7f4ae2a444aa-kube-api-access-bp7zg\") pod \"cert-manager-webhook-6486ffd997-mlcnh\" (UID: \"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.773290 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.773258 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/dbd100d1-825e-4ff4-abeb-7f4ae2a444aa-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mlcnh\" (UID: \"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.773382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.773316 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7zg\" (UniqueName: \"kubernetes.io/projected/dbd100d1-825e-4ff4-abeb-7f4ae2a444aa-kube-api-access-bp7zg\") pod \"cert-manager-webhook-6486ffd997-mlcnh\" (UID: \"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:56.873935 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:56.873865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:23:57.141636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:57.141612 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mlcnh"] Sep 10 14:23:57.144132 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:23:57.144102 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbd100d1_825e_4ff4_abeb_7f4ae2a444aa.slice/crio-08bea1e568542ea784351fd6c152a59f8d3f19fd38b660360ab211882edbf3f3 WatchSource:0}: Error finding container 08bea1e568542ea784351fd6c152a59f8d3f19fd38b660360ab211882edbf3f3: Status 404 returned error can't find the container with id 08bea1e568542ea784351fd6c152a59f8d3f19fd38b660360ab211882edbf3f3 Sep 10 14:23:57.827046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:57.827011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" event={"ID":"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa","Type":"ContainerStarted","Data":"08bea1e568542ea784351fd6c152a59f8d3f19fd38b660360ab211882edbf3f3"} Sep 10 14:23:59.053534 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:59.053498 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:59.153039 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:59.152981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:23:59.623712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:23:59.623676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:00.517835 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:00.517778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:00.577921 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:00.577844 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:00.840387 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:00.840309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" event={"ID":"dbd100d1-825e-4ff4-abeb-7f4ae2a444aa","Type":"ContainerStarted","Data":"039eb9976c62c9d974e139dfcbb0109605ecf7e55daa1fecb73bda4553707e44"} Sep 10 14:24:00.840527 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:00.840382 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:24:00.857216 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:00.857175 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" podStartSLOduration=1.724033524 podStartE2EDuration="4.857162309s" totalImagesPullingTime="3.133128785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:00.855918601 +0000 UTC m=+258.709827276" watchObservedRunningTime="2026-09-10 14:24:00.857162309 +0000 UTC m=+258.711070985" Sep 10 14:24:03.320015 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:03.319981 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:03.431022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:03.430972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:04.969450 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:04.969369 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:05.071153 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:05.071124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:06.846600 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:06.846571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mlcnh" Sep 10 14:24:07.560453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:07.560418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:23.058319 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:23.058279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:23.158030 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:23.157995 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:23.719878 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:23.719835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.150746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.150713 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk"] Sep 10 14:24:24.155593 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.155575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.158726 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.158703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p5jmf\"" Sep 10 14:24:24.159258 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.159235 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 14:24:24.159373 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.159238 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 14:24:24.159373 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.159239 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 14:24:24.159373 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.159243 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 14:24:24.171180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.171159 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk"] Sep 10 14:24:24.260066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.260000 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.270437 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.270411 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-manifests\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.270531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.270457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-tmp\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.270531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.270488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.270531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.270511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5qq5\" (UniqueName: \"kubernetes.io/projected/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-kube-api-access-w5qq5\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.270628 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.270550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.360255 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.360222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.371203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.371180 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-manifests\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.371289 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.371218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-tmp\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.371289 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.371236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.371422 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.371409 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5qq5\" (UniqueName: \"kubernetes.io/projected/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-kube-api-access-w5qq5\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.371516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.371430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.371624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.371604 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-manifests\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.373395 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.373377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-tmp\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.373704 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.373685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.373781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.373752 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-webhook-cert\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.380889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.380865 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5qq5\" (UniqueName: \"kubernetes.io/projected/b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92-kube-api-access-w5qq5\") pod \"opendatahub-operator-controller-manager-5c47c8c9b-vlqwk\" (UID: \"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92\") " pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.465475 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.465410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:24.477882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.477856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.567095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.567066 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.667382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.667348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.743904 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.743880 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk"] Sep 10 14:24:24.746719 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:24:24.746684 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cca3ab_2749_4c33_b0d7_ed2fe69bbd92.slice/crio-3c83211f987042162ccc84974dff2091ff4d4d27571e8ac23f27ef39ec78f9d3 WatchSource:0}: Error finding container 3c83211f987042162ccc84974dff2091ff4d4d27571e8ac23f27ef39ec78f9d3: Status 404 returned error can't find the container with id 3c83211f987042162ccc84974dff2091ff4d4d27571e8ac23f27ef39ec78f9d3 Sep 10 14:24:24.747919 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.747889 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.808908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.808875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.909008 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.908982 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:24.920421 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:24.920397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" event={"ID":"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92","Type":"ContainerStarted","Data":"3c83211f987042162ccc84974dff2091ff4d4d27571e8ac23f27ef39ec78f9d3"} Sep 10 14:24:27.934224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:27.934194 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" event={"ID":"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92","Type":"ContainerStarted","Data":"dd6824acc3c39629128386c06dea9ba7555a2618fca696d63c74850c6f075f31"} Sep 10 14:24:28.939239 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:28.939204 2576 generic.go:356] "Generic (PLEG): container finished" podID="b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92" containerID="dd6824acc3c39629128386c06dea9ba7555a2618fca696d63c74850c6f075f31" exitCode=0 Sep 10 14:24:28.939540 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:28.939266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" event={"ID":"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92","Type":"ContainerDied","Data":"dd6824acc3c39629128386c06dea9ba7555a2618fca696d63c74850c6f075f31"} Sep 10 14:24:29.944719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:29.944677 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" event={"ID":"b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92","Type":"ContainerStarted","Data":"d562b06ad3b5788fc8cc4634318f6b901d4ce4ee3b3639e1d967ab173a6f6fab"} Sep 10 14:24:29.945116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:29.944761 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:29.970148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:29.970088 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" podStartSLOduration=1.491382719 podStartE2EDuration="5.970073357s" totalImagesPullingTime="3.455678547s" totalInitContainerRuntime="1.023012091s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:29.967364979 +0000 UTC m=+287.821273654" watchObservedRunningTime="2026-09-10 14:24:29.970073357 +0000 UTC m=+287.823982083" Sep 10 14:24:40.950222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:40.950190 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5c47c8c9b-vlqwk" Sep 10 14:24:41.335642 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:41.335550 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:42.528135 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:42.528094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:24:42.530121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:42.530086 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:24:42.531271 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:42.531252 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 14:24:44.280736 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.280703 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp"] Sep 10 14:24:44.283736 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.283720 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.286736 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.286716 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 10 14:24:44.287286 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.287269 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 10 14:24:44.288265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.288247 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 14:24:44.288366 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.288352 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 14:24:44.300399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.300372 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp"] Sep 10 14:24:44.426923 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.426891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gtcs\" (UniqueName: \"kubernetes.io/projected/c25c4402-a1d2-4635-8806-a2a598642bfe-kube-api-access-7gtcs\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.427089 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.426950 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25c4402-a1d2-4635-8806-a2a598642bfe-tls-certs\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.427089 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.427023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c25c4402-a1d2-4635-8806-a2a598642bfe-tmp\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.527996 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.527952 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25c4402-a1d2-4635-8806-a2a598642bfe-tls-certs\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.528182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.528005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c25c4402-a1d2-4635-8806-a2a598642bfe-tmp\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.528182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.528074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gtcs\" (UniqueName: \"kubernetes.io/projected/c25c4402-a1d2-4635-8806-a2a598642bfe-kube-api-access-7gtcs\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.530240 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.530218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c25c4402-a1d2-4635-8806-a2a598642bfe-tmp\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.530364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.530345 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c25c4402-a1d2-4635-8806-a2a598642bfe-tls-certs\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.537047 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.536996 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gtcs\" (UniqueName: \"kubernetes.io/projected/c25c4402-a1d2-4635-8806-a2a598642bfe-kube-api-access-7gtcs\") pod \"kube-auth-proxy-79c7cbc6bf-2ctcp\" (UID: \"c25c4402-a1d2-4635-8806-a2a598642bfe\") " pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.593723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.593687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" Sep 10 14:24:44.865744 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.865666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp"] Sep 10 14:24:44.868549 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:24:44.868516 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25c4402_a1d2_4635_8806_a2a598642bfe.slice/crio-9a5efcb2e9c00f1b42142b945957405c053435cb9b36e84f86888f8e1bc47a86 WatchSource:0}: Error finding container 9a5efcb2e9c00f1b42142b945957405c053435cb9b36e84f86888f8e1bc47a86: Status 404 returned error can't find the container with id 9a5efcb2e9c00f1b42142b945957405c053435cb9b36e84f86888f8e1bc47a86 Sep 10 14:24:44.870370 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.870352 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:24:44.997865 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:44.997826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" event={"ID":"c25c4402-a1d2-4635-8806-a2a598642bfe","Type":"ContainerStarted","Data":"9a5efcb2e9c00f1b42142b945957405c053435cb9b36e84f86888f8e1bc47a86"} Sep 10 14:24:45.318666 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.318632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:45.663580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.663545 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv"] Sep 10 14:24:45.666620 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.666599 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.669321 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.669288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 14:24:45.669539 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.669371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 14:24:45.669539 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.669492 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 14:24:45.669724 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.669600 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xzlkm\"" Sep 10 14:24:45.669724 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.669715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 14:24:45.671692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.671625 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:45.681445 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.681414 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv"] Sep 10 14:24:45.766827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.766765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:45.840681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.840890 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpfp\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-kube-api-access-ftpfp\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.840890 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.840890 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840851 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.841061 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5281744c-ccfa-44c8-a7da-44a3e7b36923-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.841061 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.841061 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.840958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5281744c-ccfa-44c8-a7da-44a3e7b36923-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941740 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpfp\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-kube-api-access-ftpfp\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.941900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.941839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.942819 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.942772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.944775 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.944749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5281744c-ccfa-44c8-a7da-44a3e7b36923-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.944986 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.944965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.945774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.945727 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.945900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.945838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.958137 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.958111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.962418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.962374 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpfp\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-kube-api-access-ftpfp\") pod \"istiod-openshift-gateway-7b7995d5cd-w8nmv\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:45.978334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:45.978279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:46.265622 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:46.265541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv"] Sep 10 14:24:46.268083 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:24:46.268049 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5281744c_ccfa_44c8_a7da_44a3e7b36923.slice/crio-c75b3aff1df91bf60e1270918cd9826087130908b5a265e9bab59ab289be3baa WatchSource:0}: Error finding container c75b3aff1df91bf60e1270918cd9826087130908b5a265e9bab59ab289be3baa: Status 404 returned error can't find the container with id c75b3aff1df91bf60e1270918cd9826087130908b5a265e9bab59ab289be3baa Sep 10 14:24:46.677127 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:46.677086 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:47.008160 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:47.008079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" event={"ID":"5281744c-ccfa-44c8-a7da-44a3e7b36923","Type":"ContainerStarted","Data":"c75b3aff1df91bf60e1270918cd9826087130908b5a265e9bab59ab289be3baa"} Sep 10 14:24:48.015061 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:48.015020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" event={"ID":"c25c4402-a1d2-4635-8806-a2a598642bfe","Type":"ContainerStarted","Data":"e6f42f80780f6fe8411c6e9ca60abaef2fae2408be5c13cdf08153f11c4805bf"} Sep 10 14:24:48.033948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:48.033901 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-79c7cbc6bf-2ctcp" podStartSLOduration=0.983615229 podStartE2EDuration="4.03388276s" totalImagesPullingTime="3.050267531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:48.032364407 +0000 UTC m=+305.886273084" watchObservedRunningTime="2026-09-10 14:24:48.03388276 +0000 UTC m=+305.887791436" Sep 10 14:24:49.820803 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:49.820755 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:24:49.821048 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:49.820857 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:24:50.024804 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.024760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" event={"ID":"5281744c-ccfa-44c8-a7da-44a3e7b36923","Type":"ContainerStarted","Data":"c52812e9761eea0b1367e885692babb2a84e4b73c4ce44c9dc65b9cea0cdf035"} Sep 10 14:24:50.025099 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.025072 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:50.026375 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.026352 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-w8nmv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 14:24:50.026468 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.026395 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" podUID="5281744c-ccfa-44c8-a7da-44a3e7b36923" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 14:24:50.055952 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.055902 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" podStartSLOduration=1.5059871729999998 podStartE2EDuration="5.055885204s" totalImagesPullingTime="3.549898031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:50.054694503 +0000 UTC m=+307.908603179" watchObservedRunningTime="2026-09-10 14:24:50.055885204 +0000 UTC m=+307.909793879" Sep 10 14:24:50.592272 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.592234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:50.664499 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.664461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:50.892767 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.892731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:50.908442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.908415 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv"] Sep 10 14:24:50.911768 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.911748 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:50.918399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.918350 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qgtmn\"" Sep 10 14:24:50.924563 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:50.924541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv"] Sep 10 14:24:51.029159 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.029133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:24:51.034143 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.034112 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:51.089838 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.089769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.089901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131c2791-30c9-480d-9ac2-19739506c833-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.089937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.089976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131c2791-30c9-480d-9ac2-19739506c833-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.090009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131c2791-30c9-480d-9ac2-19739506c833-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.090048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.090094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.090119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q97tz\" (UniqueName: \"kubernetes.io/projected/131c2791-30c9-480d-9ac2-19739506c833-kube-api-access-q97tz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.090264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.090138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.121760 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.121727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:51.190887 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.190781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131c2791-30c9-480d-9ac2-19739506c833-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.190887 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.190862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.190887 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.190884 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131c2791-30c9-480d-9ac2-19739506c833-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.190935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131c2791-30c9-480d-9ac2-19739506c833-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.190963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.190997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q97tz\" (UniqueName: \"kubernetes.io/projected/131c2791-30c9-480d-9ac2-19739506c833-kube-api-access-q97tz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191448 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191521 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191496 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191613 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191588 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191671 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191643 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.191782 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.191756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/131c2791-30c9-480d-9ac2-19739506c833-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.193413 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.193394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/131c2791-30c9-480d-9ac2-19739506c833-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.193669 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.193650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/131c2791-30c9-480d-9ac2-19739506c833-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.199628 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.199603 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/131c2791-30c9-480d-9ac2-19739506c833-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.199720 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.199658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q97tz\" (UniqueName: \"kubernetes.io/projected/131c2791-30c9-480d-9ac2-19739506c833-kube-api-access-q97tz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv\" (UID: \"131c2791-30c9-480d-9ac2-19739506c833\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.221624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.221596 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:51.499343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:51.499318 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv"] Sep 10 14:24:51.501415 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:24:51.501384 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod131c2791_30c9_480d_9ac2_19739506c833.slice/crio-ad31b4c4eb86e8f02469cf545ae992341ad8c49dc1222e199132201f4e998d62 WatchSource:0}: Error finding container ad31b4c4eb86e8f02469cf545ae992341ad8c49dc1222e199132201f4e998d62: Status 404 returned error can't find the container with id ad31b4c4eb86e8f02469cf545ae992341ad8c49dc1222e199132201f4e998d62 Sep 10 14:24:52.038032 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:52.037995 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" event={"ID":"131c2791-30c9-480d-9ac2-19739506c833","Type":"ContainerStarted","Data":"ad31b4c4eb86e8f02469cf545ae992341ad8c49dc1222e199132201f4e998d62"} Sep 10 14:24:53.936356 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:53.936313 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:24:53.936636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:53.936395 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:24:53.936636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:53.936433 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:24:54.048560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:54.048518 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" event={"ID":"131c2791-30c9-480d-9ac2-19739506c833","Type":"ContainerStarted","Data":"c3947590c8a7c76e287cf6e77361c2699fefaf3eb5df15239b809ae8e6b4bb4a"} Sep 10 14:24:54.069447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:54.069395 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" podStartSLOduration=1.636851169 podStartE2EDuration="4.069376611s" totalImagesPullingTime="2.432525442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:54.067818037 +0000 UTC m=+311.921726713" watchObservedRunningTime="2026-09-10 14:24:54.069376611 +0000 UTC m=+311.923285288" Sep 10 14:24:54.222624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:54.222541 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:54.223846 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:54.223820 2576 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused" start-of-body= Sep 10 14:24:54.223963 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:54.223867 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" podUID="131c2791-30c9-480d-9ac2-19739506c833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.19:15021/healthz/ready\": dial tcp 10.134.0.19:15021: connect: connection refused" Sep 10 14:24:55.000834 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.000779 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:55.102082 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.102029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:55.202460 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.202403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:55.226187 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.226158 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:55.302866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.302753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:55.539767 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.539733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:55.552079 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.552008 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-phsrl"] Sep 10 14:24:55.556248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.556198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:55.558983 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.558958 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5zt2g\"" Sep 10 14:24:55.558983 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.558979 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 10 14:24:55.570514 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.570482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-phsrl"] Sep 10 14:24:55.632686 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.632650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnpz\" (UniqueName: \"kubernetes.io/projected/2fbbd3d3-18af-465f-9ce7-323d095d83cd-kube-api-access-jnnpz\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:55.632686 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.632694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbbd3d3-18af-465f-9ce7-323d095d83cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:55.733874 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.733834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnpz\" (UniqueName: \"kubernetes.io/projected/2fbbd3d3-18af-465f-9ce7-323d095d83cd-kube-api-access-jnnpz\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:55.733874 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.733877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbbd3d3-18af-465f-9ce7-323d095d83cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:55.734065 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:24:55.734038 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 10 14:24:55.734126 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:24:55.734114 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fbbd3d3-18af-465f-9ce7-323d095d83cd-tls-certs podName:2fbbd3d3-18af-465f-9ce7-323d095d83cd nodeName:}" failed. No retries permitted until 2026-09-10 14:24:56.234096521 +0000 UTC m=+314.088005188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2fbbd3d3-18af-465f-9ce7-323d095d83cd-tls-certs") pod "model-serving-api-54d89bd5fc-phsrl" (UID: "2fbbd3d3-18af-465f-9ce7-323d095d83cd") : secret "model-serving-api-tls" not found Sep 10 14:24:55.743070 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:55.743047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnpz\" (UniqueName: \"kubernetes.io/projected/2fbbd3d3-18af-465f-9ce7-323d095d83cd-kube-api-access-jnnpz\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:56.013383 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.013279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:56.056940 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.056909 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:56.057986 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.057966 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv" Sep 10 14:24:56.237689 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.237650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbbd3d3-18af-465f-9ce7-323d095d83cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:56.240015 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.239995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbbd3d3-18af-465f-9ce7-323d095d83cd-tls-certs\") pod \"model-serving-api-54d89bd5fc-phsrl\" (UID: \"2fbbd3d3-18af-465f-9ce7-323d095d83cd\") " pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:56.473058 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.473023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:56.746154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.746102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-phsrl"] Sep 10 14:24:56.748382 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:24:56.748351 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbbd3d3_18af_465f_9ce7_323d095d83cd.slice/crio-4be207774224242ffd11db6d440f6d0c7d5049a8eea869b5d709b742163d06b0 WatchSource:0}: Error finding container 4be207774224242ffd11db6d440f6d0c7d5049a8eea869b5d709b742163d06b0: Status 404 returned error can't find the container with id 4be207774224242ffd11db6d440f6d0c7d5049a8eea869b5d709b742163d06b0 Sep 10 14:24:56.996804 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:56.996685 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:57.061755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:57.061716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" event={"ID":"2fbbd3d3-18af-465f-9ce7-323d095d83cd","Type":"ContainerStarted","Data":"4be207774224242ffd11db6d440f6d0c7d5049a8eea869b5d709b742163d06b0"} Sep 10 14:24:57.130597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:57.130560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:24:59.074211 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:59.074168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" event={"ID":"2fbbd3d3-18af-465f-9ce7-323d095d83cd","Type":"ContainerStarted","Data":"6a24dacefeb0b9fe24fabdeab0d77618e6a447683497de176c42d0913c837e7f"} Sep 10 14:24:59.074690 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:59.074304 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:24:59.091380 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:24:59.091326 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" podStartSLOduration=2.391446223 podStartE2EDuration="4.091308523s" totalImagesPullingTime="1.6998623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:24:59.089837124 +0000 UTC m=+316.943745802" watchObservedRunningTime="2026-09-10 14:24:59.091308523 +0000 UTC m=+316.945217200" Sep 10 14:25:04.635544 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.635503 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:04.712611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.712572 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ggb62"] Sep 10 14:25:04.719555 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.719532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:04.726809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.726777 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-ws9hj\"" Sep 10 14:25:04.731639 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.731620 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 10 14:25:04.735753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.735731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ggb62"] Sep 10 14:25:04.751345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.751317 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h"] Sep 10 14:25:04.753570 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.753552 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.759012 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.758993 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 10 14:25:04.760204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.760104 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gxtv8\"" Sep 10 14:25:04.763206 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.763125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:04.793963 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.793925 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h"] Sep 10 14:25:04.801825 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.801785 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe5a37b4-2adc-41a4-8dee-4deac62d1414-cert\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:04.801951 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.801896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcp8d\" (UniqueName: \"kubernetes.io/projected/5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb-kube-api-access-vcp8d\") pod \"llmisvc-controller-manager-777c6bd7d5-mc25h\" (UID: \"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.801951 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.801935 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mc25h\" (UID: \"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.802043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.801992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfplz\" (UniqueName: \"kubernetes.io/projected/fe5a37b4-2adc-41a4-8dee-4deac62d1414-kube-api-access-rfplz\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:04.902855 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.902741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe5a37b4-2adc-41a4-8dee-4deac62d1414-cert\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:04.902855 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.902838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcp8d\" (UniqueName: \"kubernetes.io/projected/5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb-kube-api-access-vcp8d\") pod \"llmisvc-controller-manager-777c6bd7d5-mc25h\" (UID: \"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.903085 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.902863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mc25h\" (UID: \"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.903085 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:25:04.902892 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 10 14:25:04.903085 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:25:04.902984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe5a37b4-2adc-41a4-8dee-4deac62d1414-cert podName:fe5a37b4-2adc-41a4-8dee-4deac62d1414 nodeName:}" failed. No retries permitted until 2026-09-10 14:25:05.402966441 +0000 UTC m=+323.256875095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fe5a37b4-2adc-41a4-8dee-4deac62d1414-cert") pod "kserve-controller-manager-d6f4b65b8-ggb62" (UID: "fe5a37b4-2adc-41a4-8dee-4deac62d1414") : secret "kserve-webhook-server-cert" not found Sep 10 14:25:04.903085 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.902894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfplz\" (UniqueName: \"kubernetes.io/projected/fe5a37b4-2adc-41a4-8dee-4deac62d1414-kube-api-access-rfplz\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:04.905278 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.905259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mc25h\" (UID: \"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.912632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.912600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcp8d\" (UniqueName: \"kubernetes.io/projected/5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb-kube-api-access-vcp8d\") pod \"llmisvc-controller-manager-777c6bd7d5-mc25h\" (UID: \"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:04.912819 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:04.912786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfplz\" (UniqueName: \"kubernetes.io/projected/fe5a37b4-2adc-41a4-8dee-4deac62d1414-kube-api-access-rfplz\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:05.064939 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.064910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:05.278393 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.278309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:05.349200 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.349050 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h"] Sep 10 14:25:05.351603 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:25:05.351575 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c6048e5_38d1_4e1f_9ba2_50aa6a5e85cb.slice/crio-67a1344d60fa2e29f63384f573da025fe0e509b0ba32fc731c3cd28232e0271f WatchSource:0}: Error finding container 67a1344d60fa2e29f63384f573da025fe0e509b0ba32fc731c3cd28232e0271f: Status 404 returned error can't find the container with id 67a1344d60fa2e29f63384f573da025fe0e509b0ba32fc731c3cd28232e0271f Sep 10 14:25:05.377116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.377087 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:05.406062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.406032 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe5a37b4-2adc-41a4-8dee-4deac62d1414-cert\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:05.408393 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.408365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fe5a37b4-2adc-41a4-8dee-4deac62d1414-cert\") pod \"kserve-controller-manager-d6f4b65b8-ggb62\" (UID: \"fe5a37b4-2adc-41a4-8dee-4deac62d1414\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:05.629260 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.629228 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:05.956261 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:05.956232 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-ggb62"] Sep 10 14:25:05.956966 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:25:05.956942 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5a37b4_2adc_41a4_8dee_4deac62d1414.slice/crio-c11afb8715c2b3e8dcc650fceaa950fb8e6fdcd2b97420dc3c8232a8ceb8354d WatchSource:0}: Error finding container c11afb8715c2b3e8dcc650fceaa950fb8e6fdcd2b97420dc3c8232a8ceb8354d: Status 404 returned error can't find the container with id c11afb8715c2b3e8dcc650fceaa950fb8e6fdcd2b97420dc3c8232a8ceb8354d Sep 10 14:25:06.050733 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.050697 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:06.104212 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.104178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" event={"ID":"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb","Type":"ContainerStarted","Data":"67a1344d60fa2e29f63384f573da025fe0e509b0ba32fc731c3cd28232e0271f"} Sep 10 14:25:06.105347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.105323 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" event={"ID":"fe5a37b4-2adc-41a4-8dee-4deac62d1414","Type":"ContainerStarted","Data":"c11afb8715c2b3e8dcc650fceaa950fb8e6fdcd2b97420dc3c8232a8ceb8354d"} Sep 10 14:25:06.224175 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.224079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:06.461817 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.461761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:06.736249 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.736217 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv"] Sep 10 14:25:06.736464 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:06.736439 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" podUID="5281744c-ccfa-44c8-a7da-44a3e7b36923" containerName="discovery" containerID="cri-o://c52812e9761eea0b1367e885692babb2a84e4b73c4ce44c9dc65b9cea0cdf035" gracePeriod=30 Sep 10 14:25:07.043824 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.043737 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:07.115028 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.114990 2576 generic.go:356] "Generic (PLEG): container finished" podID="5281744c-ccfa-44c8-a7da-44a3e7b36923" containerID="c52812e9761eea0b1367e885692babb2a84e4b73c4ce44c9dc65b9cea0cdf035" exitCode=0 Sep 10 14:25:07.115173 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.115080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" event={"ID":"5281744c-ccfa-44c8-a7da-44a3e7b36923","Type":"ContainerDied","Data":"c52812e9761eea0b1367e885692babb2a84e4b73c4ce44c9dc65b9cea0cdf035"} Sep 10 14:25:07.115173 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.115125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" event={"ID":"5281744c-ccfa-44c8-a7da-44a3e7b36923","Type":"ContainerDied","Data":"c75b3aff1df91bf60e1270918cd9826087130908b5a265e9bab59ab289be3baa"} Sep 10 14:25:07.115173 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.115146 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c75b3aff1df91bf60e1270918cd9826087130908b5a265e9bab59ab289be3baa" Sep 10 14:25:07.130972 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.130857 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:25:07.219859 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.219823 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5281744c-ccfa-44c8-a7da-44a3e7b36923-local-certs\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220006 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.219874 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-ca-configmap\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220006 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.219913 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-cacerts\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220006 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.219938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-token\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220179 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.220015 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-kubeconfig\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220179 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.220033 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-dns-cert\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220179 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.220065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftpfp\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-kube-api-access-ftpfp\") pod \"5281744c-ccfa-44c8-a7da-44a3e7b36923\" (UID: \"5281744c-ccfa-44c8-a7da-44a3e7b36923\") " Sep 10 14:25:07.220537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.220484 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:25:07.222822 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.222764 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:25:07.222925 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.222864 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5281744c-ccfa-44c8-a7da-44a3e7b36923-local-certs" (OuterVolumeSpecName: "local-certs") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:25:07.223061 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.223001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:25:07.223178 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.223097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-cacerts" (OuterVolumeSpecName: "cacerts") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:25:07.223178 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.223120 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-kube-api-access-ftpfp" (OuterVolumeSpecName: "kube-api-access-ftpfp") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "kube-api-access-ftpfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:25:07.223178 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.223120 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-token" (OuterVolumeSpecName: "istio-token") pod "5281744c-ccfa-44c8-a7da-44a3e7b36923" (UID: "5281744c-ccfa-44c8-a7da-44a3e7b36923"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:25:07.307125 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.307032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:07.321142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321110 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-dns-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:07.321142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321139 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpfp\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-kube-api-access-ftpfp\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:07.321142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321149 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5281744c-ccfa-44c8-a7da-44a3e7b36923-local-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:07.321365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321158 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-csr-ca-configmap\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:07.321365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321167 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-cacerts\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:07.321365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321175 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:07.321365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:07.321184 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5281744c-ccfa-44c8-a7da-44a3e7b36923-istio-kubeconfig\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:25:08.120301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:08.120268 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv" Sep 10 14:25:08.162725 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:08.162687 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv"] Sep 10 14:25:08.178852 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:08.178818 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-w8nmv"] Sep 10 14:25:08.761012 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:08.760977 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5281744c-ccfa-44c8-a7da-44a3e7b36923" path="/var/lib/kubelet/pods/5281744c-ccfa-44c8-a7da-44a3e7b36923/volumes" Sep 10 14:25:10.084415 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.084380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-phsrl" Sep 10 14:25:10.132010 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.131979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" event={"ID":"5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb","Type":"ContainerStarted","Data":"422445a24294a1bd567b8013652010998bdacc4c098d650afda41251197d7226"} Sep 10 14:25:10.132177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.132073 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:10.133837 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.133781 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" event={"ID":"fe5a37b4-2adc-41a4-8dee-4deac62d1414","Type":"ContainerStarted","Data":"01d482a2b1f2b3c18dea5e2ac319b63f2b4485e36222aed4a81aa62958924bd6"} Sep 10 14:25:10.133962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.133921 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:10.153940 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.153864 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" podStartSLOduration=2.412005624 podStartE2EDuration="6.153845837s" totalImagesPullingTime="3.741840213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:10.152638158 +0000 UTC m=+328.006546835" watchObservedRunningTime="2026-09-10 14:25:10.153845837 +0000 UTC m=+328.007754514" Sep 10 14:25:10.176828 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.176765 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" podStartSLOduration=3.038401394 podStartE2EDuration="6.176753093s" totalImagesPullingTime="3.138351699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:25:10.17466936 +0000 UTC m=+328.028578036" watchObservedRunningTime="2026-09-10 14:25:10.176753093 +0000 UTC m=+328.030661768" Sep 10 14:25:10.437250 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:10.437211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:13.081564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:13.081534 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:13.144590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:13.144556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:20.172620 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:20.172575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:41.140818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:41.140726 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mc25h" Sep 10 14:25:41.143568 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:41.143547 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-ggb62" Sep 10 14:25:56.040832 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:56.040783 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:25:56.139779 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:25:56.139739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:10.769929 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:10.769896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:12.816752 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.816718 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6"] Sep 10 14:26:12.817223 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.817017 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5281744c-ccfa-44c8-a7da-44a3e7b36923" containerName="discovery" Sep 10 14:26:12.817223 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.817029 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5281744c-ccfa-44c8-a7da-44a3e7b36923" containerName="discovery" Sep 10 14:26:12.817223 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.817088 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5281744c-ccfa-44c8-a7da-44a3e7b36923" containerName="discovery" Sep 10 14:26:12.819903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.819887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:12.822862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.822834 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 14:26:12.822862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.822834 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-rr92z\"" Sep 10 14:26:12.824032 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.824014 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 14:26:12.833706 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.833683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6"] Sep 10 14:26:12.846549 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.846523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4h84\" (UniqueName: \"kubernetes.io/projected/55ae7cc8-7dae-42ec-8e71-99423c0ab45b-kube-api-access-j4h84\") pod \"limitador-operator-controller-manager-74495c69f8-2qmx6\" (UID: \"55ae7cc8-7dae-42ec-8e71-99423c0ab45b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:12.947092 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.947053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4h84\" (UniqueName: \"kubernetes.io/projected/55ae7cc8-7dae-42ec-8e71-99423c0ab45b-kube-api-access-j4h84\") pod \"limitador-operator-controller-manager-74495c69f8-2qmx6\" (UID: \"55ae7cc8-7dae-42ec-8e71-99423c0ab45b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:12.955638 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:12.955607 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4h84\" (UniqueName: \"kubernetes.io/projected/55ae7cc8-7dae-42ec-8e71-99423c0ab45b-kube-api-access-j4h84\") pod \"limitador-operator-controller-manager-74495c69f8-2qmx6\" (UID: \"55ae7cc8-7dae-42ec-8e71-99423c0ab45b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:13.129086 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:13.129061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:13.401068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:13.400887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6"] Sep 10 14:26:13.403954 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:13.403925 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ae7cc8_7dae_42ec_8e71_99423c0ab45b.slice/crio-ae00c27c3ccf544b42652a54efd40f9a61229232d485dde9b26b2be8a5096cc4 WatchSource:0}: Error finding container ae00c27c3ccf544b42652a54efd40f9a61229232d485dde9b26b2be8a5096cc4: Status 404 returned error can't find the container with id ae00c27c3ccf544b42652a54efd40f9a61229232d485dde9b26b2be8a5096cc4 Sep 10 14:26:13.578265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:13.578229 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:14.395677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.395634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" event={"ID":"55ae7cc8-7dae-42ec-8e71-99423c0ab45b","Type":"ContainerStarted","Data":"ae00c27c3ccf544b42652a54efd40f9a61229232d485dde9b26b2be8a5096cc4"} Sep 10 14:26:14.820225 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.820145 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f"] Sep 10 14:26:14.823537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.823516 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:14.826915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.826891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 10 14:26:14.827398 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.827371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tl5z5\"" Sep 10 14:26:14.837985 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.837958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f"] Sep 10 14:26:14.864395 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.864362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvq65\" (UniqueName: \"kubernetes.io/projected/e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf-kube-api-access-wvq65\") pod \"dns-operator-controller-manager-7b84cd7c9c-rvp7f\" (UID: \"e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:14.965922 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.965885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvq65\" (UniqueName: \"kubernetes.io/projected/e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf-kube-api-access-wvq65\") pod \"dns-operator-controller-manager-7b84cd7c9c-rvp7f\" (UID: \"e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:14.974486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:14.974460 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvq65\" (UniqueName: \"kubernetes.io/projected/e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf-kube-api-access-wvq65\") pod \"dns-operator-controller-manager-7b84cd7c9c-rvp7f\" (UID: \"e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:15.131413 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.131379 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-556fc4b877-89zkr"] Sep 10 14:26:15.135685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.135655 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:15.137240 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.137210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.146457 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.146214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556fc4b877-89zkr"] Sep 10 14:26:15.166807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.166769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-oauth-config\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.166948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.166886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gfzv\" (UniqueName: \"kubernetes.io/projected/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-kube-api-access-7gfzv\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.166948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.166923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-serving-cert\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.167050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.166957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-oauth-serving-cert\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.167050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.166982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-config\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.167050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.167032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-service-ca\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.167167 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.167059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-trusted-ca-bundle\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.188823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.188769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:15.267811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267755 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-oauth-config\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.267987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gfzv\" (UniqueName: \"kubernetes.io/projected/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-kube-api-access-7gfzv\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.267987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-serving-cert\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.267987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-oauth-serving-cert\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.267987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267906 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-config\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.267987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-service-ca\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.267987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.267963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-trusted-ca-bundle\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.268820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.268720 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-oauth-serving-cert\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.268820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.268768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-config\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.268995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.268945 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-service-ca\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.269053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.269028 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-trusted-ca-bundle\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.270589 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.270561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-oauth-config\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.270752 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.270731 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-console-serving-cert\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.276530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.276508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gfzv\" (UniqueName: \"kubernetes.io/projected/d3a0c1fe-aa59-4b47-a0f5-61c9733c9862-kube-api-access-7gfzv\") pod \"console-556fc4b877-89zkr\" (UID: \"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862\") " pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.297911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.297878 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:15.449854 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.449754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:15.574813 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.574679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:15.818282 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:15.818191 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:16.357932 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.357879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556fc4b877-89zkr"] Sep 10 14:26:16.360465 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:16.360429 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3a0c1fe_aa59_4b47_a0f5_61c9733c9862.slice/crio-cb63a45e647d507e5f9ab6a92ad2562c7995d3a148a40dcadfadcabdd04832f4 WatchSource:0}: Error finding container cb63a45e647d507e5f9ab6a92ad2562c7995d3a148a40dcadfadcabdd04832f4: Status 404 returned error can't find the container with id cb63a45e647d507e5f9ab6a92ad2562c7995d3a148a40dcadfadcabdd04832f4 Sep 10 14:26:16.361087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.361067 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f"] Sep 10 14:26:16.364054 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:16.364018 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e0cc5d_bd53_43ce_bae9_75bce3a41bbf.slice/crio-55d3ebdbe7720f7621799b180ac77933cff71e54b9eddb911a3edb69f24e6829 WatchSource:0}: Error finding container 55d3ebdbe7720f7621799b180ac77933cff71e54b9eddb911a3edb69f24e6829: Status 404 returned error can't find the container with id 55d3ebdbe7720f7621799b180ac77933cff71e54b9eddb911a3edb69f24e6829 Sep 10 14:26:16.407005 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.406979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" event={"ID":"e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf","Type":"ContainerStarted","Data":"55d3ebdbe7720f7621799b180ac77933cff71e54b9eddb911a3edb69f24e6829"} Sep 10 14:26:16.408207 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.408179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556fc4b877-89zkr" event={"ID":"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862","Type":"ContainerStarted","Data":"cb63a45e647d507e5f9ab6a92ad2562c7995d3a148a40dcadfadcabdd04832f4"} Sep 10 14:26:16.409718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.409692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" event={"ID":"55ae7cc8-7dae-42ec-8e71-99423c0ab45b","Type":"ContainerStarted","Data":"b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a"} Sep 10 14:26:16.409841 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.409823 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:16.429541 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:16.429491 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" podStartSLOduration=1.683480726 podStartE2EDuration="4.429476099s" totalImagesPullingTime="2.745995373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:16.429108212 +0000 UTC m=+394.283016891" watchObservedRunningTime="2026-09-10 14:26:16.429476099 +0000 UTC m=+394.283384775" Sep 10 14:26:17.224741 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.224706 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:17.416166 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.416121 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556fc4b877-89zkr" event={"ID":"d3a0c1fe-aa59-4b47-a0f5-61c9733c9862","Type":"ContainerStarted","Data":"d5ec08e461361cb6790ab2a575533e584990e57a3d20624ed5334081f75c6b6d"} Sep 10 14:26:17.445624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.445568 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-556fc4b877-89zkr" podStartSLOduration=2.4455512710000002 podStartE2EDuration="2.445551271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:17.443540912 +0000 UTC m=+395.297449588" watchObservedRunningTime="2026-09-10 14:26:17.445551271 +0000 UTC m=+395.299459946" Sep 10 14:26:17.538402 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.538321 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mft7v"] Sep 10 14:26:17.541766 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.541743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:17.545562 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.545497 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vv2kp\"" Sep 10 14:26:17.558655 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.558595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mft7v"] Sep 10 14:26:17.591054 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.591025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdq8g\" (UniqueName: \"kubernetes.io/projected/6329f60d-daeb-4fa8-81e9-493efb21f2ee-kube-api-access-hdq8g\") pod \"authorino-operator-76b478bd78-mft7v\" (UID: \"6329f60d-daeb-4fa8-81e9-493efb21f2ee\") " pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:17.691843 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.691786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdq8g\" (UniqueName: \"kubernetes.io/projected/6329f60d-daeb-4fa8-81e9-493efb21f2ee-kube-api-access-hdq8g\") pod \"authorino-operator-76b478bd78-mft7v\" (UID: \"6329f60d-daeb-4fa8-81e9-493efb21f2ee\") " pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:17.703130 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.703103 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdq8g\" (UniqueName: \"kubernetes.io/projected/6329f60d-daeb-4fa8-81e9-493efb21f2ee-kube-api-access-hdq8g\") pod \"authorino-operator-76b478bd78-mft7v\" (UID: \"6329f60d-daeb-4fa8-81e9-493efb21f2ee\") " pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:17.854637 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:17.854558 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:18.132106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.131988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-mft7v"] Sep 10 14:26:18.189596 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.189392 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:18.286068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.286012 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:18.341232 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:18.341198 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6329f60d_daeb_4fa8_81e9_493efb21f2ee.slice/crio-9446321e157525d90049162ad1f79cd142e2e346ea415e6e729f65563bff5815 WatchSource:0}: Error finding container 9446321e157525d90049162ad1f79cd142e2e346ea415e6e729f65563bff5815: Status 404 returned error can't find the container with id 9446321e157525d90049162ad1f79cd142e2e346ea415e6e729f65563bff5815 Sep 10 14:26:18.421994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.421965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" event={"ID":"6329f60d-daeb-4fa8-81e9-493efb21f2ee","Type":"ContainerStarted","Data":"9446321e157525d90049162ad1f79cd142e2e346ea415e6e729f65563bff5815"} Sep 10 14:26:18.562068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.562021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:18.665372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.665283 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:18.960819 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:18.960722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:19.061705 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:19.061674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:19.428005 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:19.427974 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" event={"ID":"e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf","Type":"ContainerStarted","Data":"50a1d3679b85d0157c172308ca7f1409a3e199bde2a0ad366d99a6e4a602e6d8"} Sep 10 14:26:19.428374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:19.428107 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:19.445561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:19.445509 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" podStartSLOduration=3.414321613 podStartE2EDuration="5.445495089s" totalImagesPullingTime="2.031173476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:19.444032648 +0000 UTC m=+397.297941324" watchObservedRunningTime="2026-09-10 14:26:19.445495089 +0000 UTC m=+397.299403764" Sep 10 14:26:19.596526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:19.596489 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:19.701670 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:19.701575 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:20.435430 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:20.435388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" event={"ID":"6329f60d-daeb-4fa8-81e9-493efb21f2ee","Type":"ContainerStarted","Data":"636277fe86c1da4aeedb08a23f3b4f6c976c5f770826a8ab23fa94254c9a447d"} Sep 10 14:26:20.460900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:20.460851 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" podStartSLOduration=1.689042752 podStartE2EDuration="3.460835675s" totalImagesPullingTime="1.771792923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:20.459158523 +0000 UTC m=+398.313067200" watchObservedRunningTime="2026-09-10 14:26:20.460835675 +0000 UTC m=+398.314744351" Sep 10 14:26:21.439528 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:21.439498 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:22.185832 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.185786 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g"] Sep 10 14:26:22.189123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.189107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.192118 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.192097 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zk7bh\"" Sep 10 14:26:22.220568 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.220538 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g"] Sep 10 14:26:22.232313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.232287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/276df055-12a4-4301-915e-9256ba7e98b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.232550 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.232523 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstqs\" (UniqueName: \"kubernetes.io/projected/276df055-12a4-4301-915e-9256ba7e98b2-kube-api-access-nstqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.333232 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.333207 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/276df055-12a4-4301-915e-9256ba7e98b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.333349 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.333243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstqs\" (UniqueName: \"kubernetes.io/projected/276df055-12a4-4301-915e-9256ba7e98b2-kube-api-access-nstqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.333572 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.333553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/276df055-12a4-4301-915e-9256ba7e98b2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.344566 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.344540 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstqs\" (UniqueName: \"kubernetes.io/projected/276df055-12a4-4301-915e-9256ba7e98b2-kube-api-access-nstqs\") pod \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.498650 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.498566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:22.775137 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:22.775113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g"] Sep 10 14:26:22.777179 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:22.777152 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276df055_12a4_4301_915e_9256ba7e98b2.slice/crio-ecd749ad012c2ce8e3319d00419bd2c9308486bf3a955614370634cdcbb1f49a WatchSource:0}: Error finding container ecd749ad012c2ce8e3319d00419bd2c9308486bf3a955614370634cdcbb1f49a: Status 404 returned error can't find the container with id ecd749ad012c2ce8e3319d00419bd2c9308486bf3a955614370634cdcbb1f49a Sep 10 14:26:23.450659 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:23.450628 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" event={"ID":"276df055-12a4-4301-915e-9256ba7e98b2","Type":"ContainerStarted","Data":"ecd749ad012c2ce8e3319d00419bd2c9308486bf3a955614370634cdcbb1f49a"} Sep 10 14:26:23.470046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:23.470009 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:23.569671 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:23.569605 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:25.450740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.450702 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:25.451179 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.450763 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:25.457556 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.457533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:25.464193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.464171 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-556fc4b877-89zkr" Sep 10 14:26:25.527174 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.527138 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd74ccb95-xzskc"] Sep 10 14:26:25.660842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.660755 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:25.761676 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.761589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:25.833181 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:25.833134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:26.185528 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:26.185490 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:26.285284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:26.285230 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:27.418742 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:27.418709 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:27.471302 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:27.471266 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" event={"ID":"276df055-12a4-4301-915e-9256ba7e98b2","Type":"ContainerStarted","Data":"e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8"} Sep 10 14:26:27.471507 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:27.471409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:27.495761 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:27.495704 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" podStartSLOduration=1.8256877729999998 podStartE2EDuration="5.495684672s" totalImagesPullingTime="3.669996899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:27.492676555 +0000 UTC m=+405.346585230" watchObservedRunningTime="2026-09-10 14:26:27.495684672 +0000 UTC m=+405.349593351" Sep 10 14:26:29.098014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:29.097984 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:29.499336 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:29.499302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:30.438001 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:30.437972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rvp7f" Sep 10 14:26:31.228165 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:31.228130 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:31.328723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:31.328686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:32.445557 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:32.445526 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-mft7v" Sep 10 14:26:35.959245 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:35.959210 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:36.060288 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:36.060251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:36.159162 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:36.159123 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:38.477946 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:38.477910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:39.827097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:39.827048 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f"] Sep 10 14:26:39.832949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:39.832930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:39.846994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:39.846966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f"] Sep 10 14:26:39.979994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:39.979961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:39.980177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:39.980008 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vgc\" (UniqueName: \"kubernetes.io/projected/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-kube-api-access-58vgc\") pod \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.081014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.080918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.081014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.080985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vgc\" (UniqueName: \"kubernetes.io/projected/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-kube-api-access-58vgc\") pod \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.081362 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.081334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.097887 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.097857 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vgc\" (UniqueName: \"kubernetes.io/projected/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-kube-api-access-58vgc\") pod \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.142429 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.142391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.420377 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.420327 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f"] Sep 10 14:26:40.423561 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:40.423534 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0faeb6a5_4e84_491f_b0f1_1a0f8f54cabe.slice/crio-0719d0617a93128688b5c4dadbadcde4bf0a084f3d1c2bea088f7d60129d5701 WatchSource:0}: Error finding container 0719d0617a93128688b5c4dadbadcde4bf0a084f3d1c2bea088f7d60129d5701: Status 404 returned error can't find the container with id 0719d0617a93128688b5c4dadbadcde4bf0a084f3d1c2bea088f7d60129d5701 Sep 10 14:26:40.531123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.531083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" event={"ID":"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe","Type":"ContainerStarted","Data":"b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d"} Sep 10 14:26:40.531259 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.531143 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.531259 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.531160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" event={"ID":"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe","Type":"ContainerStarted","Data":"0719d0617a93128688b5c4dadbadcde4bf0a084f3d1c2bea088f7d60129d5701"} Sep 10 14:26:40.540224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.540194 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f"] Sep 10 14:26:40.542704 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.542652 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"lastTransitionTime\\\":\\\"2026-09-10T14:26:40Z\\\",\\\"status\\\":\\\"True\\\",\\\"type\\\":\\\"PodReadyToStartContainers\\\"}],\\\"containerStatuses\\\":[{\\\"allocatedResources\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"},\\\"containerID\\\":\\\"cri-o://b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d\\\",\\\"image\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:cc2471e083a549afb80327aa87e1d32f05e40561c3f82275b0cdb88af5ee5e8e\\\",\\\"imageID\\\":\\\"registry.redhat.io/rhcl-1/rhcl-rhel9-operator@sha256:22341c0a543abdcd6cecb58e0e84ffeb713799db22bdf3c01e38cf5c248aee0e\\\",\\\"lastState\\\":{},\\\"name\\\":\\\"manager\\\",\\\"ready\\\":false,\\\"resources\\\":{\\\"limits\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"300Mi\\\"},\\\"requests\\\":{\\\"cpu\\\":\\\"200m\\\",\\\"memory\\\":\\\"200Mi\\\"}},\\\"restartCount\\\":0,\\\"started\\\":true,\\\"state\\\":{\\\"running\\\":{\\\"startedAt\\\":\\\"2026-09-10T14:26:40Z\\\"}},\\\"user\\\":{\\\"linux\\\":{\\\"gid\\\":0,\\\"supplementalGroups\\\":[0,1000680000],\\\"uid\\\":1000680000}},\\\"volumeMounts\\\":[{\\\"mountPath\\\":\\\"/tmp/kuadrant\\\",\\\"name\\\":\\\"extensions-socket-volume\\\"},{\\\"mountPath\\\":\\\"/var/run/secrets/kubernetes.io/serviceaccount\\\",\\\"name\\\":\\\"kube-api-access-58vgc\\\",\\\"readOnly\\\":true,\\\"recursiveReadOnly\\\":\\\"Disabled\\\"}]}],\\\"phase\\\":\\\"Running\\\",\\\"podIP\\\":\\\"10.134.0.28\\\",\\\"podIPs\\\":[{\\\"ip\\\":\\\"10.134.0.28\\\"}]}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-5f685c496f-vmr5f\": pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.543703 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.543681 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f"] Sep 10 14:26:40.567954 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.567920 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw"] Sep 10 14:26:40.568466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.568446 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" containerName="manager" Sep 10 14:26:40.568519 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.568469 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" containerName="manager" Sep 10 14:26:40.568554 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.568541 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" containerName="manager" Sep 10 14:26:40.572075 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.572056 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.575955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.575933 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g"] Sep 10 14:26:40.576107 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.576087 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" podUID="276df055-12a4-4301-915e-9256ba7e98b2" containerName="manager" containerID="cri-o://e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8" gracePeriod=2 Sep 10 14:26:40.585074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.585018 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g"] Sep 10 14:26:40.592405 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.592370 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw"] Sep 10 14:26:40.607402 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.607373 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6"] Sep 10 14:26:40.607590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.607569 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" containerName="manager" containerID="cri-o://b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a" gracePeriod=2 Sep 10 14:26:40.622687 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.622651 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6"] Sep 10 14:26:40.648223 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.648181 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.650702 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.650674 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.655310 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.655274 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-2qmx6\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:40.657888 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.657855 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="276df055-12a4-4301-915e-9256ba7e98b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:40.665811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.665773 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4"] Sep 10 14:26:40.666101 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.666089 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" containerName="manager" Sep 10 14:26:40.666156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.666102 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" containerName="manager" Sep 10 14:26:40.666156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.666120 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276df055-12a4-4301-915e-9256ba7e98b2" containerName="manager" Sep 10 14:26:40.666156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.666125 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="276df055-12a4-4301-915e-9256ba7e98b2" containerName="manager" Sep 10 14:26:40.666242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.666171 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="276df055-12a4-4301-915e-9256ba7e98b2" containerName="manager" Sep 10 14:26:40.666242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.666182 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" containerName="manager" Sep 10 14:26:40.669272 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.669229 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:40.671991 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.671936 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-2qmx6\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:40.684330 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.684304 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4"] Sep 10 14:26:40.685833 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.685808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa46ea90-9469-4993-81d0-2430e4341d1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-srmsw\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.685965 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.685868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28xv\" (UniqueName: \"kubernetes.io/projected/aa46ea90-9469-4993-81d0-2430e4341d1a-kube-api-access-n28xv\") pod \"kuadrant-operator-controller-manager-5f685c496f-srmsw\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.703699 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.703671 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-ccq4g\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="276df055-12a4-4301-915e-9256ba7e98b2" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:40.705888 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:40.705869 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:40.787189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.787161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa46ea90-9469-4993-81d0-2430e4341d1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-srmsw\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.787313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.787204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28xv\" (UniqueName: \"kubernetes.io/projected/aa46ea90-9469-4993-81d0-2430e4341d1a-kube-api-access-n28xv\") pod \"kuadrant-operator-controller-manager-5f685c496f-srmsw\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.787313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.787268 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz6dx\" (UniqueName: \"kubernetes.io/projected/ad0fc8cb-763d-4bbd-9865-168871b6286c-kube-api-access-vz6dx\") pod \"limitador-operator-controller-manager-74495c69f8-dgdc4\" (UID: \"ad0fc8cb-763d-4bbd-9865-168871b6286c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:40.787570 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.787547 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa46ea90-9469-4993-81d0-2430e4341d1a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-srmsw\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.795889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.795866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28xv\" (UniqueName: \"kubernetes.io/projected/aa46ea90-9469-4993-81d0-2430e4341d1a-kube-api-access-n28xv\") pod \"kuadrant-operator-controller-manager-5f685c496f-srmsw\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.828624 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.828576 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:40.835220 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.835202 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:40.884431 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.884390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:40.888688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.888660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz6dx\" (UniqueName: \"kubernetes.io/projected/ad0fc8cb-763d-4bbd-9865-168871b6286c-kube-api-access-vz6dx\") pod \"limitador-operator-controller-manager-74495c69f8-dgdc4\" (UID: \"ad0fc8cb-763d-4bbd-9865-168871b6286c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:40.901071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.901048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz6dx\" (UniqueName: \"kubernetes.io/projected/ad0fc8cb-763d-4bbd-9865-168871b6286c-kube-api-access-vz6dx\") pod \"limitador-operator-controller-manager-74495c69f8-dgdc4\" (UID: \"ad0fc8cb-763d-4bbd-9865-168871b6286c\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:40.989428 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.989394 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nstqs\" (UniqueName: \"kubernetes.io/projected/276df055-12a4-4301-915e-9256ba7e98b2-kube-api-access-nstqs\") pod \"276df055-12a4-4301-915e-9256ba7e98b2\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " Sep 10 14:26:40.989564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.989472 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4h84\" (UniqueName: \"kubernetes.io/projected/55ae7cc8-7dae-42ec-8e71-99423c0ab45b-kube-api-access-j4h84\") pod \"55ae7cc8-7dae-42ec-8e71-99423c0ab45b\" (UID: \"55ae7cc8-7dae-42ec-8e71-99423c0ab45b\") " Sep 10 14:26:40.989564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.989503 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/276df055-12a4-4301-915e-9256ba7e98b2-extensions-socket-volume\") pod \"276df055-12a4-4301-915e-9256ba7e98b2\" (UID: \"276df055-12a4-4301-915e-9256ba7e98b2\") " Sep 10 14:26:40.990051 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.990015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276df055-12a4-4301-915e-9256ba7e98b2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "276df055-12a4-4301-915e-9256ba7e98b2" (UID: "276df055-12a4-4301-915e-9256ba7e98b2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:26:40.991574 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.991553 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ae7cc8-7dae-42ec-8e71-99423c0ab45b-kube-api-access-j4h84" (OuterVolumeSpecName: "kube-api-access-j4h84") pod "55ae7cc8-7dae-42ec-8e71-99423c0ab45b" (UID: "55ae7cc8-7dae-42ec-8e71-99423c0ab45b"). InnerVolumeSpecName "kube-api-access-j4h84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:26:40.991623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.991585 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276df055-12a4-4301-915e-9256ba7e98b2-kube-api-access-nstqs" (OuterVolumeSpecName: "kube-api-access-nstqs") pod "276df055-12a4-4301-915e-9256ba7e98b2" (UID: "276df055-12a4-4301-915e-9256ba7e98b2"). InnerVolumeSpecName "kube-api-access-nstqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:26:40.993700 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:40.993683 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:41.090405 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.090376 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4h84\" (UniqueName: \"kubernetes.io/projected/55ae7cc8-7dae-42ec-8e71-99423c0ab45b-kube-api-access-j4h84\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:41.090405 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.090403 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/276df055-12a4-4301-915e-9256ba7e98b2-extensions-socket-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:41.090578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.090412 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nstqs\" (UniqueName: \"kubernetes.io/projected/276df055-12a4-4301-915e-9256ba7e98b2-kube-api-access-nstqs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:41.158064 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.158037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw"] Sep 10 14:26:41.160475 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:41.160452 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa46ea90_9469_4993_81d0_2430e4341d1a.slice/crio-7140bccf7996bc359d265099fbabba2d2d1ddb6b58b35f8768fa81984f8ed479 WatchSource:0}: Error finding container 7140bccf7996bc359d265099fbabba2d2d1ddb6b58b35f8768fa81984f8ed479: Status 404 returned error can't find the container with id 7140bccf7996bc359d265099fbabba2d2d1ddb6b58b35f8768fa81984f8ed479 Sep 10 14:26:41.280296 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.280267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4"] Sep 10 14:26:41.283727 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:41.283698 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0fc8cb_763d_4bbd_9865_168871b6286c.slice/crio-7401acb1a5cedd3873f3d7e4ded1c5707d28f3e32ecaf25dd3d409b621a6b3c8 WatchSource:0}: Error finding container 7401acb1a5cedd3873f3d7e4ded1c5707d28f3e32ecaf25dd3d409b621a6b3c8: Status 404 returned error can't find the container with id 7401acb1a5cedd3873f3d7e4ded1c5707d28f3e32ecaf25dd3d409b621a6b3c8 Sep 10 14:26:41.536852 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.536745 2576 generic.go:356] "Generic (PLEG): container finished" podID="276df055-12a4-4301-915e-9256ba7e98b2" containerID="e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8" exitCode=0 Sep 10 14:26:41.536852 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.536830 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-ccq4g" Sep 10 14:26:41.537043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.536851 2576 scope.go:117] "RemoveContainer" containerID="e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8" Sep 10 14:26:41.538391 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.538364 2576 generic.go:356] "Generic (PLEG): container finished" podID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" containerID="b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a" exitCode=0 Sep 10 14:26:41.538519 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.538427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2qmx6" Sep 10 14:26:41.540474 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.540429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" event={"ID":"ad0fc8cb-763d-4bbd-9865-168871b6286c","Type":"ContainerStarted","Data":"d4a2dd453ee2960d8b932c7789636e9894469db022291e049722c5304e6b3d86"} Sep 10 14:26:41.540697 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.540675 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:41.540809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.540711 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" event={"ID":"ad0fc8cb-763d-4bbd-9865-168871b6286c","Type":"ContainerStarted","Data":"7401acb1a5cedd3873f3d7e4ded1c5707d28f3e32ecaf25dd3d409b621a6b3c8"} Sep 10 14:26:41.542475 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.542450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" event={"ID":"aa46ea90-9469-4993-81d0-2430e4341d1a","Type":"ContainerStarted","Data":"31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da"} Sep 10 14:26:41.542587 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.542537 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:41.542587 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.542579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" event={"ID":"aa46ea90-9469-4993-81d0-2430e4341d1a","Type":"ContainerStarted","Data":"7140bccf7996bc359d265099fbabba2d2d1ddb6b58b35f8768fa81984f8ed479"} Sep 10 14:26:41.542702 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:41.542687 2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" container="manager" Sep 10 14:26:41.547543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.547510 2576 scope.go:117] "RemoveContainer" containerID="e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8" Sep 10 14:26:41.547849 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:41.547787 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8\": container with ID starting with e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8 not found: ID does not exist" containerID="e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8" Sep 10 14:26:41.547924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.547854 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8"} err="failed to get container status \"e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8\": rpc error: code = NotFound desc = could not find container \"e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8\": container with ID starting with e4443e38986078d5629b2a428d036369f3e98ac3d3c99bbe20dc0fe45380a9c8 not found: ID does not exist" Sep 10 14:26:41.547924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.547870 2576 scope.go:117] "RemoveContainer" containerID="b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a" Sep 10 14:26:41.557370 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.557347 2576 scope.go:117] "RemoveContainer" containerID="b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a" Sep 10 14:26:41.557776 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:41.557752 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a\": container with ID starting with b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a not found: ID does not exist" containerID="b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a" Sep 10 14:26:41.557915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.557782 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a"} err="failed to get container status \"b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a\": rpc error: code = NotFound desc = could not find container \"b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a\": container with ID starting with b91e5c51974a2b5886020a5f824daa784141587adb35ae4f78ebcf83054b572a not found: ID does not exist" Sep 10 14:26:41.564640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.564601 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" podStartSLOduration=1.564589784 podStartE2EDuration="1.564589784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:41.562391778 +0000 UTC m=+419.416300454" watchObservedRunningTime="2026-09-10 14:26:41.564589784 +0000 UTC m=+419.418498459" Sep 10 14:26:41.608557 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:41.608515 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" podStartSLOduration=1.6085019200000001 podStartE2EDuration="1.60850192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:41.605767351 +0000 UTC m=+419.459676027" watchObservedRunningTime="2026-09-10 14:26:41.60850192 +0000 UTC m=+419.462410596" Sep 10 14:26:42.549981 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.549925 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" containerName="manager" containerID="cri-o://b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d" gracePeriod=2 Sep 10 14:26:42.552319 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:42.552292 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:42.758988 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:42.758955 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:42.761333 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.761307 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276df055-12a4-4301-915e-9256ba7e98b2" path="/var/lib/kubelet/pods/276df055-12a4-4301-915e-9256ba7e98b2/volumes" Sep 10 14:26:42.761639 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.761625 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ae7cc8-7dae-42ec-8e71-99423c0ab45b" path="/var/lib/kubelet/pods/55ae7cc8-7dae-42ec-8e71-99423c0ab45b/volumes" Sep 10 14:26:42.792128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.792084 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:42.839109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.839080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:42.907001 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.906972 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-extensions-socket-volume\") pod \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " Sep 10 14:26:42.907148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.907080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58vgc\" (UniqueName: \"kubernetes.io/projected/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-kube-api-access-58vgc\") pod \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\" (UID: \"0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe\") " Sep 10 14:26:42.907241 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.907218 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" (UID: "0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:26:42.907317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.907304 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-extensions-socket-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:42.909074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.909049 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-kube-api-access-58vgc" (OuterVolumeSpecName: "kube-api-access-58vgc") pod "0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" (UID: "0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe"). InnerVolumeSpecName "kube-api-access-58vgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:26:42.938438 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:42.938404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:43.008526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:43.008497 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58vgc\" (UniqueName: \"kubernetes.io/projected/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe-kube-api-access-58vgc\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:43.555623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:43.555590 2576 generic.go:356] "Generic (PLEG): container finished" podID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" containerID="b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d" exitCode=2 Sep 10 14:26:43.555995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:43.555633 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:43.555995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:43.555684 2576 scope.go:117] "RemoveContainer" containerID="b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d" Sep 10 14:26:43.558343 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:43.558320 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:43.564247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:43.564229 2576 scope.go:117] "RemoveContainer" containerID="b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d" Sep 10 14:26:43.564508 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:43.564489 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d\": container with ID starting with b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d not found: ID does not exist" containerID="b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d" Sep 10 14:26:43.564570 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:43.564513 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d"} err="failed to get container status \"b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d\": rpc error: code = NotFound desc = could not find container \"b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d\": container with ID starting with b34dd789d8559b255a3a4c27fd5a0f0186145b8904fe0cf8bfa53c4cfc67dc8d not found: ID does not exist" Sep 10 14:26:43.566107 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:43.566088 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-vmr5f\" is forbidden: User \"system:node:ip-10-0-135-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-236.ec2.internal' and this object" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-vmr5f" Sep 10 14:26:44.760247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:44.760215 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe" path="/var/lib/kubelet/pods/0faeb6a5-4e84-491f-b0f1-1a0f8f54cabe/volumes" Sep 10 14:26:45.784277 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:45.784240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:45.881813 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:45.881758 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:50.555808 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.555742 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6bd74ccb95-xzskc" podUID="997f1b20-c8e5-4679-9e70-b5ee59e1771f" containerName="console" containerID="cri-o://fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59" gracePeriod=15 Sep 10 14:26:50.797479 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.797453 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:26:50.877014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.876991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-service-ca\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877025 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-trusted-ca-bundle\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877057 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-serving-cert\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877095 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-oauth-serving-cert\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877160 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-config\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877204 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-oauth-config\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877237 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7brn2\" (UniqueName: \"kubernetes.io/projected/997f1b20-c8e5-4679-9e70-b5ee59e1771f-kube-api-access-7brn2\") pod \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\" (UID: \"997f1b20-c8e5-4679-9e70-b5ee59e1771f\") " Sep 10 14:26:50.877563 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:26:50.877563 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877526 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-service-ca" (OuterVolumeSpecName: "service-ca") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:26:50.877770 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877665 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-config" (OuterVolumeSpecName: "console-config") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:26:50.877770 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.877722 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:26:50.879359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.879336 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:26:50.879359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.879349 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997f1b20-c8e5-4679-9e70-b5ee59e1771f-kube-api-access-7brn2" (OuterVolumeSpecName: "kube-api-access-7brn2") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "kube-api-access-7brn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:26:50.879610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.879404 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "997f1b20-c8e5-4679-9e70-b5ee59e1771f" (UID: "997f1b20-c8e5-4679-9e70-b5ee59e1771f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:26:50.924379 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.924349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:50.978193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978167 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-serving-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.978193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978188 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-oauth-serving-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.978193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978198 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-config\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.978371 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978206 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/997f1b20-c8e5-4679-9e70-b5ee59e1771f-console-oauth-config\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.978371 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978216 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7brn2\" (UniqueName: \"kubernetes.io/projected/997f1b20-c8e5-4679-9e70-b5ee59e1771f-kube-api-access-7brn2\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.978371 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978224 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.978371 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.978232 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/997f1b20-c8e5-4679-9e70-b5ee59e1771f-trusted-ca-bundle\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:26:50.986894 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:50.986868 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:51.600318 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.600283 2576 generic.go:356] "Generic (PLEG): container finished" podID="997f1b20-c8e5-4679-9e70-b5ee59e1771f" containerID="fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59" exitCode=0 Sep 10 14:26:51.600748 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.600324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd74ccb95-xzskc" event={"ID":"997f1b20-c8e5-4679-9e70-b5ee59e1771f","Type":"ContainerDied","Data":"fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59"} Sep 10 14:26:51.600748 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.600360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bd74ccb95-xzskc" event={"ID":"997f1b20-c8e5-4679-9e70-b5ee59e1771f","Type":"ContainerDied","Data":"f339bea517b9703dcc6fe8fb51a1065668b4709b6c80dddfb64b9fa7ce90c8cb"} Sep 10 14:26:51.600748 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.600379 2576 scope.go:117] "RemoveContainer" containerID="fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59" Sep 10 14:26:51.600748 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.600327 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bd74ccb95-xzskc" Sep 10 14:26:51.614377 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.614358 2576 scope.go:117] "RemoveContainer" containerID="fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59" Sep 10 14:26:51.614626 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:26:51.614607 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59\": container with ID starting with fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59 not found: ID does not exist" containerID="fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59" Sep 10 14:26:51.614667 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.614634 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59"} err="failed to get container status \"fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59\": rpc error: code = NotFound desc = could not find container \"fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59\": container with ID starting with fa0088c642bc89a411814f1057472894ce14c3682c68dc6aa10310c1b82f2d59 not found: ID does not exist" Sep 10 14:26:51.625267 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.625246 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bd74ccb95-xzskc"] Sep 10 14:26:51.628718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:51.628697 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bd74ccb95-xzskc"] Sep 10 14:26:52.116305 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.116273 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:52.174581 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.174549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:52.552273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.552246 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dgdc4" Sep 10 14:26:52.552449 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.552314 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:26:52.759525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.759491 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="997f1b20-c8e5-4679-9e70-b5ee59e1771f" path="/var/lib/kubelet/pods/997f1b20-c8e5-4679-9e70-b5ee59e1771f/volumes" Sep 10 14:26:52.859567 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.859539 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c"] Sep 10 14:26:52.859881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.859868 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="997f1b20-c8e5-4679-9e70-b5ee59e1771f" containerName="console" Sep 10 14:26:52.859934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.859883 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="997f1b20-c8e5-4679-9e70-b5ee59e1771f" containerName="console" Sep 10 14:26:52.859968 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.859953 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="997f1b20-c8e5-4679-9e70-b5ee59e1771f" containerName="console" Sep 10 14:26:52.862720 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.862704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:52.889522 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.889497 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c"] Sep 10 14:26:52.891309 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.891285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwz45\" (UniqueName: \"kubernetes.io/projected/7eaccd13-9f96-447c-a418-82b10db5750e-kube-api-access-lwz45\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kql7c\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:52.891430 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.891322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eaccd13-9f96-447c-a418-82b10db5750e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kql7c\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:52.992262 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.992234 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwz45\" (UniqueName: \"kubernetes.io/projected/7eaccd13-9f96-447c-a418-82b10db5750e-kube-api-access-lwz45\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kql7c\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:52.992408 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.992274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eaccd13-9f96-447c-a418-82b10db5750e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kql7c\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:52.992579 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:52.992563 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eaccd13-9f96-447c-a418-82b10db5750e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kql7c\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:53.003640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.003619 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwz45\" (UniqueName: \"kubernetes.io/projected/7eaccd13-9f96-447c-a418-82b10db5750e-kube-api-access-lwz45\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kql7c\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:53.172457 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.172428 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:53.447520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.447443 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c"] Sep 10 14:26:53.451202 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:26:53.451178 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaccd13_9f96_447c_a418_82b10db5750e.slice/crio-eb1ef1e330210cb45da4d2ad10f7d5366f25d824dcdcf483d06cbf47833187fb WatchSource:0}: Error finding container eb1ef1e330210cb45da4d2ad10f7d5366f25d824dcdcf483d06cbf47833187fb: Status 404 returned error can't find the container with id eb1ef1e330210cb45da4d2ad10f7d5366f25d824dcdcf483d06cbf47833187fb Sep 10 14:26:53.614020 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.613976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" event={"ID":"7eaccd13-9f96-447c-a418-82b10db5750e","Type":"ContainerStarted","Data":"ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c"} Sep 10 14:26:53.614020 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.614024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" event={"ID":"7eaccd13-9f96-447c-a418-82b10db5750e","Type":"ContainerStarted","Data":"eb1ef1e330210cb45da4d2ad10f7d5366f25d824dcdcf483d06cbf47833187fb"} Sep 10 14:26:53.614231 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.614110 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:26:53.641715 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:53.641670 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" podStartSLOduration=1.641657135 podStartE2EDuration="1.641657135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:26:53.640176845 +0000 UTC m=+431.494085523" watchObservedRunningTime="2026-09-10 14:26:53.641657135 +0000 UTC m=+431.495565810" Sep 10 14:26:58.322934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:58.322900 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:58.384591 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:58.384553 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:58.924066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:58.924034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:59.022242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:59.022200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:59.521974 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:59.521939 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:59.584758 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:59.584730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:59.686695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:59.686660 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:26:59.787930 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:26:59.787859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:04.621432 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.621402 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:27:04.671336 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.671304 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw"] Sep 10 14:27:04.671733 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.671698 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" podUID="aa46ea90-9469-4993-81d0-2430e4341d1a" containerName="manager" containerID="cri-o://31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da" gracePeriod=10 Sep 10 14:27:04.916354 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.916332 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:27:04.971745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.971714 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa46ea90-9469-4993-81d0-2430e4341d1a-extensions-socket-volume\") pod \"aa46ea90-9469-4993-81d0-2430e4341d1a\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " Sep 10 14:27:04.971922 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.971753 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n28xv\" (UniqueName: \"kubernetes.io/projected/aa46ea90-9469-4993-81d0-2430e4341d1a-kube-api-access-n28xv\") pod \"aa46ea90-9469-4993-81d0-2430e4341d1a\" (UID: \"aa46ea90-9469-4993-81d0-2430e4341d1a\") " Sep 10 14:27:04.972084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.972059 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa46ea90-9469-4993-81d0-2430e4341d1a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aa46ea90-9469-4993-81d0-2430e4341d1a" (UID: "aa46ea90-9469-4993-81d0-2430e4341d1a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:27:04.973810 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:04.973778 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa46ea90-9469-4993-81d0-2430e4341d1a-kube-api-access-n28xv" (OuterVolumeSpecName: "kube-api-access-n28xv") pod "aa46ea90-9469-4993-81d0-2430e4341d1a" (UID: "aa46ea90-9469-4993-81d0-2430e4341d1a"). InnerVolumeSpecName "kube-api-access-n28xv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:27:05.073177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.073148 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aa46ea90-9469-4993-81d0-2430e4341d1a-extensions-socket-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:27:05.073177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.073175 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n28xv\" (UniqueName: \"kubernetes.io/projected/aa46ea90-9469-4993-81d0-2430e4341d1a-kube-api-access-n28xv\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:27:05.677066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.677033 2576 generic.go:356] "Generic (PLEG): container finished" podID="aa46ea90-9469-4993-81d0-2430e4341d1a" containerID="31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da" exitCode=0 Sep 10 14:27:05.677520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.677074 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" Sep 10 14:27:05.677520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.677110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" event={"ID":"aa46ea90-9469-4993-81d0-2430e4341d1a","Type":"ContainerDied","Data":"31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da"} Sep 10 14:27:05.677520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.677165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw" event={"ID":"aa46ea90-9469-4993-81d0-2430e4341d1a","Type":"ContainerDied","Data":"7140bccf7996bc359d265099fbabba2d2d1ddb6b58b35f8768fa81984f8ed479"} Sep 10 14:27:05.677520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.677201 2576 scope.go:117] "RemoveContainer" containerID="31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da" Sep 10 14:27:05.685779 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.685761 2576 scope.go:117] "RemoveContainer" containerID="31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da" Sep 10 14:27:05.686072 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:27:05.686053 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da\": container with ID starting with 31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da not found: ID does not exist" containerID="31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da" Sep 10 14:27:05.686123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.686080 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da"} err="failed to get container status \"31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da\": rpc error: code = NotFound desc = could not find container \"31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da\": container with ID starting with 31ec2cce17b3134b8ae3b34be7a6d3638a8e630673b0dfb5a7358d4fcc32f9da not found: ID does not exist" Sep 10 14:27:05.703004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.702965 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw"] Sep 10 14:27:05.707008 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:05.706988 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-srmsw"] Sep 10 14:27:06.760031 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:06.760000 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa46ea90-9469-4993-81d0-2430e4341d1a" path="/var/lib/kubelet/pods/aa46ea90-9469-4993-81d0-2430e4341d1a/volumes" Sep 10 14:27:11.514608 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:11.514563 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:11.651380 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:11.651348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:23.785431 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:23.785389 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:23.890367 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:23.890334 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:23.945424 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:23.945393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:24.161549 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.161495 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:27:24.161831 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.161819 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa46ea90-9469-4993-81d0-2430e4341d1a" containerName="manager" Sep 10 14:27:24.161882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.161833 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa46ea90-9469-4993-81d0-2430e4341d1a" containerName="manager" Sep 10 14:27:24.161921 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.161902 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa46ea90-9469-4993-81d0-2430e4341d1a" containerName="manager" Sep 10 14:27:24.164809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.164770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.167659 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.167629 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 10 14:27:24.167659 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.167649 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-drgvk\"" Sep 10 14:27:24.175109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.175088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:27:24.203661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.203631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:27:24.225641 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.225615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl5fz\" (UniqueName: \"kubernetes.io/projected/4f220e00-79a0-4497-bc8c-aaf8865eeedd-kube-api-access-dl5fz\") pod \"limitador-limitador-84994f7ffc-x9gtg\" (UID: \"4f220e00-79a0-4497-bc8c-aaf8865eeedd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.225739 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.225649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f220e00-79a0-4497-bc8c-aaf8865eeedd-config-file\") pod \"limitador-limitador-84994f7ffc-x9gtg\" (UID: \"4f220e00-79a0-4497-bc8c-aaf8865eeedd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.326938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.326908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl5fz\" (UniqueName: \"kubernetes.io/projected/4f220e00-79a0-4497-bc8c-aaf8865eeedd-kube-api-access-dl5fz\") pod \"limitador-limitador-84994f7ffc-x9gtg\" (UID: \"4f220e00-79a0-4497-bc8c-aaf8865eeedd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.327050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.326942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f220e00-79a0-4497-bc8c-aaf8865eeedd-config-file\") pod \"limitador-limitador-84994f7ffc-x9gtg\" (UID: \"4f220e00-79a0-4497-bc8c-aaf8865eeedd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.327546 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.327527 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4f220e00-79a0-4497-bc8c-aaf8865eeedd-config-file\") pod \"limitador-limitador-84994f7ffc-x9gtg\" (UID: \"4f220e00-79a0-4497-bc8c-aaf8865eeedd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.334534 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.334508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl5fz\" (UniqueName: \"kubernetes.io/projected/4f220e00-79a0-4497-bc8c-aaf8865eeedd-kube-api-access-dl5fz\") pod \"limitador-limitador-84994f7ffc-x9gtg\" (UID: \"4f220e00-79a0-4497-bc8c-aaf8865eeedd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.475520 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.475447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:24.574020 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.573988 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6vs9"] Sep 10 14:27:24.577211 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.577192 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:24.579812 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.579769 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ghtw8\"" Sep 10 14:27:24.590231 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.590208 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6vs9"] Sep 10 14:27:24.605087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.605063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:24.629642 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.629602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wj95\" (UniqueName: \"kubernetes.io/projected/32392a5c-71fe-4c6a-a691-1291b2198286-kube-api-access-2wj95\") pod \"authorino-7bcfd8f9cd-b6vs9\" (UID: \"32392a5c-71fe-4c6a-a691-1291b2198286\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:24.677776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.677751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:24.730408 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.730351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wj95\" (UniqueName: \"kubernetes.io/projected/32392a5c-71fe-4c6a-a691-1291b2198286-kube-api-access-2wj95\") pod \"authorino-7bcfd8f9cd-b6vs9\" (UID: \"32392a5c-71fe-4c6a-a691-1291b2198286\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:24.739427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.739402 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wj95\" (UniqueName: \"kubernetes.io/projected/32392a5c-71fe-4c6a-a691-1291b2198286-kube-api-access-2wj95\") pod \"authorino-7bcfd8f9cd-b6vs9\" (UID: \"32392a5c-71fe-4c6a-a691-1291b2198286\") " pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:24.747136 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.747115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:27:24.749165 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:27:24.749145 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f220e00_79a0_4497_bc8c_aaf8865eeedd.slice/crio-006941bda991c8e1c8692cb4dae9f1128b7b9dcb037f1bd01e4147827c0d31b5 WatchSource:0}: Error finding container 006941bda991c8e1c8692cb4dae9f1128b7b9dcb037f1bd01e4147827c0d31b5: Status 404 returned error can't find the container with id 006941bda991c8e1c8692cb4dae9f1128b7b9dcb037f1bd01e4147827c0d31b5 Sep 10 14:27:24.768605 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.768576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" event={"ID":"4f220e00-79a0-4497-bc8c-aaf8865eeedd","Type":"ContainerStarted","Data":"006941bda991c8e1c8692cb4dae9f1128b7b9dcb037f1bd01e4147827c0d31b5"} Sep 10 14:27:24.886374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:24.886345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:25.152968 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:25.152939 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6vs9"] Sep 10 14:27:25.155942 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:27:25.155919 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32392a5c_71fe_4c6a_a691_1291b2198286.slice/crio-05b8fb0677a4f2f3276fbd37272640665190e7d8ab23b30acbadf695d12be005 WatchSource:0}: Error finding container 05b8fb0677a4f2f3276fbd37272640665190e7d8ab23b30acbadf695d12be005: Status 404 returned error can't find the container with id 05b8fb0677a4f2f3276fbd37272640665190e7d8ab23b30acbadf695d12be005 Sep 10 14:27:25.774678 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:25.774642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" event={"ID":"32392a5c-71fe-4c6a-a691-1291b2198286","Type":"ContainerStarted","Data":"05b8fb0677a4f2f3276fbd37272640665190e7d8ab23b30acbadf695d12be005"} Sep 10 14:27:26.781543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:26.781504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" event={"ID":"4f220e00-79a0-4497-bc8c-aaf8865eeedd","Type":"ContainerStarted","Data":"630540d20d400cb41933c8e507473e7beefc924219e5cd4a2ef36faf83fa267c"} Sep 10 14:27:26.781962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:26.781580 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:26.799326 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:26.799280 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" podStartSLOduration=1.584068534 podStartE2EDuration="2.799263222s" totalImagesPullingTime="1.215194688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:26.797423527 +0000 UTC m=+464.651332204" watchObservedRunningTime="2026-09-10 14:27:26.799263222 +0000 UTC m=+464.653171903" Sep 10 14:27:27.588907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:27.588834 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6vs9"] Sep 10 14:27:27.788042 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:27.788007 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" event={"ID":"32392a5c-71fe-4c6a-a691-1291b2198286","Type":"ContainerStarted","Data":"797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de"} Sep 10 14:27:27.803034 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:27.802988 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" podStartSLOduration=1.795085804 podStartE2EDuration="3.802972307s" totalImagesPullingTime="2.007886503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:27:27.801610999 +0000 UTC m=+465.655519675" watchObservedRunningTime="2026-09-10 14:27:27.802972307 +0000 UTC m=+465.656880983" Sep 10 14:27:28.792640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:28.792598 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" podUID="32392a5c-71fe-4c6a-a691-1291b2198286" containerName="authorino" containerID="cri-o://797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de" gracePeriod=30 Sep 10 14:27:29.031942 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.031920 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:29.167343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.167311 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wj95\" (UniqueName: \"kubernetes.io/projected/32392a5c-71fe-4c6a-a691-1291b2198286-kube-api-access-2wj95\") pod \"32392a5c-71fe-4c6a-a691-1291b2198286\" (UID: \"32392a5c-71fe-4c6a-a691-1291b2198286\") " Sep 10 14:27:29.169246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.169222 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32392a5c-71fe-4c6a-a691-1291b2198286-kube-api-access-2wj95" (OuterVolumeSpecName: "kube-api-access-2wj95") pod "32392a5c-71fe-4c6a-a691-1291b2198286" (UID: "32392a5c-71fe-4c6a-a691-1291b2198286"). InnerVolumeSpecName "kube-api-access-2wj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:27:29.267881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.267848 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wj95\" (UniqueName: \"kubernetes.io/projected/32392a5c-71fe-4c6a-a691-1291b2198286-kube-api-access-2wj95\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:27:29.797993 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.797958 2576 generic.go:356] "Generic (PLEG): container finished" podID="32392a5c-71fe-4c6a-a691-1291b2198286" containerID="797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de" exitCode=0 Sep 10 14:27:29.798357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.797998 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" Sep 10 14:27:29.798357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.798011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" event={"ID":"32392a5c-71fe-4c6a-a691-1291b2198286","Type":"ContainerDied","Data":"797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de"} Sep 10 14:27:29.798357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.798040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-b6vs9" event={"ID":"32392a5c-71fe-4c6a-a691-1291b2198286","Type":"ContainerDied","Data":"05b8fb0677a4f2f3276fbd37272640665190e7d8ab23b30acbadf695d12be005"} Sep 10 14:27:29.798357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.798056 2576 scope.go:117] "RemoveContainer" containerID="797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de" Sep 10 14:27:29.806964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.806943 2576 scope.go:117] "RemoveContainer" containerID="797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de" Sep 10 14:27:29.807224 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:27:29.807206 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de\": container with ID starting with 797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de not found: ID does not exist" containerID="797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de" Sep 10 14:27:29.807269 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.807233 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de"} err="failed to get container status \"797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de\": rpc error: code = NotFound desc = could not find container \"797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de\": container with ID starting with 797afac44fbee36c86570228f1e780a22d6701a37ec5ac46736f6d5e5a0a25de not found: ID does not exist" Sep 10 14:27:29.818558 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.818537 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6vs9"] Sep 10 14:27:29.820419 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:29.820392 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-b6vs9"] Sep 10 14:27:30.760211 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:30.760179 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32392a5c-71fe-4c6a-a691-1291b2198286" path="/var/lib/kubelet/pods/32392a5c-71fe-4c6a-a691-1291b2198286/volumes" Sep 10 14:27:37.789389 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:37.789357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-x9gtg" Sep 10 14:27:39.609257 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:39.609211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:39.741914 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:39.741875 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:39.991889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:39.991851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:52.643021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.642962 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:52.811893 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.811860 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:27:52.935856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.935774 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p8h89"] Sep 10 14:27:52.936125 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.936114 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32392a5c-71fe-4c6a-a691-1291b2198286" containerName="authorino" Sep 10 14:27:52.936167 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.936128 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="32392a5c-71fe-4c6a-a691-1291b2198286" containerName="authorino" Sep 10 14:27:52.936201 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.936189 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="32392a5c-71fe-4c6a-a691-1291b2198286" containerName="authorino" Sep 10 14:27:52.940755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.940737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:52.944924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.944771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 14:27:52.944924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.944923 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ghtw8\"" Sep 10 14:27:52.947055 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:52.947027 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p8h89"] Sep 10 14:27:53.056043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.056000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq6br\" (UniqueName: \"kubernetes.io/projected/4434e319-b11c-42a1-9448-b2bf54ff496c-kube-api-access-wq6br\") pod \"authorino-6d4dfc649f-p8h89\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.056214 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.056104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4434e319-b11c-42a1-9448-b2bf54ff496c-tls-cert\") pod \"authorino-6d4dfc649f-p8h89\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.094672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.094641 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p8h89"] Sep 10 14:27:53.094899 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:27:53.094879 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wq6br tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-p8h89" podUID="4434e319-b11c-42a1-9448-b2bf54ff496c" Sep 10 14:27:53.157374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.157339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4434e319-b11c-42a1-9448-b2bf54ff496c-tls-cert\") pod \"authorino-6d4dfc649f-p8h89\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.157522 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.157395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq6br\" (UniqueName: \"kubernetes.io/projected/4434e319-b11c-42a1-9448-b2bf54ff496c-kube-api-access-wq6br\") pod \"authorino-6d4dfc649f-p8h89\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.159803 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.159771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4434e319-b11c-42a1-9448-b2bf54ff496c-tls-cert\") pod \"authorino-6d4dfc649f-p8h89\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.165084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.165061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq6br\" (UniqueName: \"kubernetes.io/projected/4434e319-b11c-42a1-9448-b2bf54ff496c-kube-api-access-wq6br\") pod \"authorino-6d4dfc649f-p8h89\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.901135 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.901102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:53.908082 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:53.908061 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:54.064253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.064225 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4434e319-b11c-42a1-9448-b2bf54ff496c-tls-cert\") pod \"4434e319-b11c-42a1-9448-b2bf54ff496c\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " Sep 10 14:27:54.064440 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.064308 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wq6br\" (UniqueName: \"kubernetes.io/projected/4434e319-b11c-42a1-9448-b2bf54ff496c-kube-api-access-wq6br\") pod \"4434e319-b11c-42a1-9448-b2bf54ff496c\" (UID: \"4434e319-b11c-42a1-9448-b2bf54ff496c\") " Sep 10 14:27:54.066388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.066364 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4434e319-b11c-42a1-9448-b2bf54ff496c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4434e319-b11c-42a1-9448-b2bf54ff496c" (UID: "4434e319-b11c-42a1-9448-b2bf54ff496c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:27:54.066482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.066413 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4434e319-b11c-42a1-9448-b2bf54ff496c-kube-api-access-wq6br" (OuterVolumeSpecName: "kube-api-access-wq6br") pod "4434e319-b11c-42a1-9448-b2bf54ff496c" (UID: "4434e319-b11c-42a1-9448-b2bf54ff496c"). InnerVolumeSpecName "kube-api-access-wq6br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:27:54.165547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.165481 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4434e319-b11c-42a1-9448-b2bf54ff496c-tls-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:27:54.165547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.165507 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq6br\" (UniqueName: \"kubernetes.io/projected/4434e319-b11c-42a1-9448-b2bf54ff496c-kube-api-access-wq6br\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:27:54.905663 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.905625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-p8h89" Sep 10 14:27:54.932827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.932800 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p8h89"] Sep 10 14:27:54.936657 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:54.936631 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-p8h89"] Sep 10 14:27:56.759744 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:27:56.759707 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4434e319-b11c-42a1-9448-b2bf54ff496c" path="/var/lib/kubelet/pods/4434e319-b11c-42a1-9448-b2bf54ff496c/volumes" Sep 10 14:28:00.194986 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:00.194954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:00.300534 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:00.300500 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:00.458972 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:00.458874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:00.534431 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:00.534398 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:00.609158 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:00.609125 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:00.849881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:00.849784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:01.045911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:01.045877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:04.626745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:04.626705 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:04.726547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:04.726483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:36.240956 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:36.240767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:36.340163 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:36.340124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:37.733817 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:37.733767 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:37.831269 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:37.831233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:37.931186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:37.931146 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:38.060811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:38.060701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:38.488903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:38.488872 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:38.582660 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:38.582611 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:28:38.592083 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:38.592039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:47.936244 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:47.936205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:48.041888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:48.041849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:57.043004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:57.042966 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:28:57.601612 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:28:57.601579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:29:06.359801 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.359763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:29:06.908462 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.908425 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf"] Sep 10 14:29:06.911806 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.911775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:06.914677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.914655 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 14:29:06.914828 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.914656 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2pwwc\"" Sep 10 14:29:06.914828 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.914660 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 14:29:06.915701 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.915683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 10 14:29:06.922248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.922227 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf"] Sep 10 14:29:06.940745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:06.940713 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:29:07.041818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.041757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.041978 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.041821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.041978 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.041908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9aa1d6-64d9-4634-a30f-575d8d97298f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.041978 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.041949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.041978 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.041976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.042155 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.042016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgq42\" (UniqueName: \"kubernetes.io/projected/ba9aa1d6-64d9-4634-a30f-575d8d97298f-kube-api-access-sgq42\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.142610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.142577 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9aa1d6-64d9-4634-a30f-575d8d97298f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.142781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.142617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.142781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.142643 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.142781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.142661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgq42\" (UniqueName: \"kubernetes.io/projected/ba9aa1d6-64d9-4634-a30f-575d8d97298f-kube-api-access-sgq42\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.142781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.142688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.143021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.142810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.143131 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.143107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.143248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.143180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.143248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.143222 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.144860 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.144840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba9aa1d6-64d9-4634-a30f-575d8d97298f-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.145110 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.145094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba9aa1d6-64d9-4634-a30f-575d8d97298f-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.151136 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.151111 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgq42\" (UniqueName: \"kubernetes.io/projected/ba9aa1d6-64d9-4634-a30f-575d8d97298f-kube-api-access-sgq42\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kkrqf\" (UID: \"ba9aa1d6-64d9-4634-a30f-575d8d97298f\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.223830 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.223750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:07.496862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.496837 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf"] Sep 10 14:29:07.498958 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:29:07.498933 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9aa1d6_64d9_4634_a30f_575d8d97298f.slice/crio-43d3b1c71382923ff3cc12b8ca8add269d321bbbdd701126354e8fc2a922f1c7 WatchSource:0}: Error finding container 43d3b1c71382923ff3cc12b8ca8add269d321bbbdd701126354e8fc2a922f1c7: Status 404 returned error can't find the container with id 43d3b1c71382923ff3cc12b8ca8add269d321bbbdd701126354e8fc2a922f1c7 Sep 10 14:29:07.704026 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:07.703996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:29:08.232161 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:08.232122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" event={"ID":"ba9aa1d6-64d9-4634-a30f-575d8d97298f","Type":"ContainerStarted","Data":"43d3b1c71382923ff3cc12b8ca8add269d321bbbdd701126354e8fc2a922f1c7"} Sep 10 14:29:09.238231 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:09.238197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" event={"ID":"ba9aa1d6-64d9-4634-a30f-575d8d97298f","Type":"ContainerStarted","Data":"49e279e246405a411f3845114fcbda6a344ab3f92bce935458db0938a53c1596"} Sep 10 14:29:09.259010 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:09.258958 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" podStartSLOduration=2.22985216 podStartE2EDuration="3.258943632s" totalImagesPullingTime="1.029091472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:09.257052077 +0000 UTC m=+567.110960792" watchObservedRunningTime="2026-09-10 14:29:09.258943632 +0000 UTC m=+567.112852308" Sep 10 14:29:16.737727 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:16.737695 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:29:17.224487 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:17.224453 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:17.224664 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:17.224510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:17.236854 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:17.236829 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:17.284861 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:17.284836 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kkrqf" Sep 10 14:29:17.707025 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:17.706995 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:29:22.911451 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.911422 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k"] Sep 10 14:29:22.914987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.914968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:22.917545 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.917521 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 14:29:22.925078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.925056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k"] Sep 10 14:29:22.978988 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.978964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:22.979111 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.979004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:22.979111 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.979024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:22.979205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.979149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533f97d3-4209-4c81-b8b1-f0984f175644-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:22.979205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.979179 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:22.979205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:22.979196 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8khcs\" (UniqueName: \"kubernetes.io/projected/533f97d3-4209-4c81-b8b1-f0984f175644-kube-api-access-8khcs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.026664 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.026630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:29:23.080252 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533f97d3-4209-4c81-b8b1-f0984f175644-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080252 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8khcs\" (UniqueName: \"kubernetes.io/projected/533f97d3-4209-4c81-b8b1-f0984f175644-kube-api-access-8khcs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080703 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080677 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080775 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080721 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.080863 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.080784 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.082499 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.082480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533f97d3-4209-4c81-b8b1-f0984f175644-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.082853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.082838 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533f97d3-4209-4c81-b8b1-f0984f175644-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.090266 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.090244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8khcs\" (UniqueName: \"kubernetes.io/projected/533f97d3-4209-4c81-b8b1-f0984f175644-kube-api-access-8khcs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-r7z8k\" (UID: \"533f97d3-4209-4c81-b8b1-f0984f175644\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.225269 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.225189 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:23.502718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:23.501305 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k"] Sep 10 14:29:23.508446 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:29:23.508415 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533f97d3_4209_4c81_b8b1_f0984f175644.slice/crio-5743192e3e1a91b767135d4290698b808a50b4bb813203bf0c024322c4298644 WatchSource:0}: Error finding container 5743192e3e1a91b767135d4290698b808a50b4bb813203bf0c024322c4298644: Status 404 returned error can't find the container with id 5743192e3e1a91b767135d4290698b808a50b4bb813203bf0c024322c4298644 Sep 10 14:29:24.307863 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:24.307826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" event={"ID":"533f97d3-4209-4c81-b8b1-f0984f175644","Type":"ContainerStarted","Data":"38dd14b9d2ee35f831f4dd488da4810100a71be9825f55323cd254b57e0922ad"} Sep 10 14:29:24.308281 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:24.307883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" event={"ID":"533f97d3-4209-4c81-b8b1-f0984f175644","Type":"ContainerStarted","Data":"5743192e3e1a91b767135d4290698b808a50b4bb813203bf0c024322c4298644"} Sep 10 14:29:24.309043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:24.309019 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:29:33.225437 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:33.225405 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:33.225862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:33.225687 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:33.237866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:33.237844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:33.256115 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:33.256075 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" podStartSLOduration=11.025925958 podStartE2EDuration="11.256062343s" totalImagesPullingTime="230.136385ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:24.328891265 +0000 UTC m=+582.182799941" watchObservedRunningTime="2026-09-10 14:29:33.256062343 +0000 UTC m=+591.109971018" Sep 10 14:29:33.363547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:33.363522 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-r7z8k" Sep 10 14:29:42.560536 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:42.560505 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:29:42.561029 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:42.560934 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:29:42.921361 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:42.921325 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:29:44.505019 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:44.504984 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:29:51.104298 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.104266 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg"] Sep 10 14:29:51.107008 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.106987 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.109669 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.109647 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 14:29:51.118941 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.118918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg"] Sep 10 14:29:51.137649 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.137622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:29:51.210818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.210772 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5893093a-c52e-40a5-9c5f-8e681e7c1956-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.210962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.210836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.210962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.210880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.210962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.210917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpzr\" (UniqueName: \"kubernetes.io/projected/5893093a-c52e-40a5-9c5f-8e681e7c1956-kube-api-access-6qpzr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.210962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.210957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.211114 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.210973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.311654 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.311613 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5893093a-c52e-40a5-9c5f-8e681e7c1956-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.311871 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.311668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.311871 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.311829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.311998 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.311870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpzr\" (UniqueName: \"kubernetes.io/projected/5893093a-c52e-40a5-9c5f-8e681e7c1956-kube-api-access-6qpzr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.311998 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.311935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.311998 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.311962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.312157 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.312085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.312197 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.312159 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.312275 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.312251 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.313985 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.313958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5893093a-c52e-40a5-9c5f-8e681e7c1956-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.314375 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.314352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5893093a-c52e-40a5-9c5f-8e681e7c1956-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.320100 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.320080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpzr\" (UniqueName: \"kubernetes.io/projected/5893093a-c52e-40a5-9c5f-8e681e7c1956-kube-api-access-6qpzr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bf2tg\" (UID: \"5893093a-c52e-40a5-9c5f-8e681e7c1956\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.417907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.417868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:29:51.688944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.688918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg"] Sep 10 14:29:51.690805 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:29:51.690765 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5893093a_c52e_40a5_9c5f_8e681e7c1956.slice/crio-393103f78eef7caf252754c9e9ebdd0ed8ba33fa4d5e2a1ed3b40ba666898b00 WatchSource:0}: Error finding container 393103f78eef7caf252754c9e9ebdd0ed8ba33fa4d5e2a1ed3b40ba666898b00: Status 404 returned error can't find the container with id 393103f78eef7caf252754c9e9ebdd0ed8ba33fa4d5e2a1ed3b40ba666898b00 Sep 10 14:29:51.692094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:51.692077 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:29:52.433718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:52.433672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" event={"ID":"5893093a-c52e-40a5-9c5f-8e681e7c1956","Type":"ContainerStarted","Data":"32cda4e871ea484fe5c91e2762c2083ec7c43c8bcd240e214f430653fcdeef22"} Sep 10 14:29:52.434093 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:52.433739 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" event={"ID":"5893093a-c52e-40a5-9c5f-8e681e7c1956","Type":"ContainerStarted","Data":"393103f78eef7caf252754c9e9ebdd0ed8ba33fa4d5e2a1ed3b40ba666898b00"} Sep 10 14:29:52.452297 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:52.452249 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" podStartSLOduration=1.260104082 podStartE2EDuration="1.452232703s" totalImagesPullingTime="192.128621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:29:52.450334854 +0000 UTC m=+610.304243531" watchObservedRunningTime="2026-09-10 14:29:52.452232703 +0000 UTC m=+610.306141379" Sep 10 14:29:52.807231 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:52.807157 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:29:54.937420 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:29:54.937385 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:30:01.418851 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:01.418813 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:30:01.419237 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:01.418878 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:30:01.431596 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:01.431571 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:30:01.486027 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:01.485993 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bf2tg" Sep 10 14:30:01.616354 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:01.616311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:30:11.774342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:11.774261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:30:11.828200 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:11.828167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:30:44.761944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:44.761913 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:30:44.821572 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:30:44.821542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:05.868001 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:05.867969 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d445769f-zlsf6"] Sep 10 14:31:05.871466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:05.871445 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:05.874148 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:05.874123 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ghtw8\"" Sep 10 14:31:05.875361 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:05.875342 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 10 14:31:05.880435 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:05.880415 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d445769f-zlsf6"] Sep 10 14:31:06.039018 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.038982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlhh\" (UniqueName: \"kubernetes.io/projected/369aaedc-43e4-4994-9bfe-7fd9e95b1f69-kube-api-access-2rlhh\") pod \"authorino-75d445769f-zlsf6\" (UID: \"369aaedc-43e4-4994-9bfe-7fd9e95b1f69\") " pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.039183 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.039036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/369aaedc-43e4-4994-9bfe-7fd9e95b1f69-tls-cert\") pod \"authorino-75d445769f-zlsf6\" (UID: \"369aaedc-43e4-4994-9bfe-7fd9e95b1f69\") " pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.140221 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.140186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/369aaedc-43e4-4994-9bfe-7fd9e95b1f69-tls-cert\") pod \"authorino-75d445769f-zlsf6\" (UID: \"369aaedc-43e4-4994-9bfe-7fd9e95b1f69\") " pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.140373 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.140275 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlhh\" (UniqueName: \"kubernetes.io/projected/369aaedc-43e4-4994-9bfe-7fd9e95b1f69-kube-api-access-2rlhh\") pod \"authorino-75d445769f-zlsf6\" (UID: \"369aaedc-43e4-4994-9bfe-7fd9e95b1f69\") " pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.142620 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.142595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/369aaedc-43e4-4994-9bfe-7fd9e95b1f69-tls-cert\") pod \"authorino-75d445769f-zlsf6\" (UID: \"369aaedc-43e4-4994-9bfe-7fd9e95b1f69\") " pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.148947 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.148929 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlhh\" (UniqueName: \"kubernetes.io/projected/369aaedc-43e4-4994-9bfe-7fd9e95b1f69-kube-api-access-2rlhh\") pod \"authorino-75d445769f-zlsf6\" (UID: \"369aaedc-43e4-4994-9bfe-7fd9e95b1f69\") " pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.180658 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.180633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d445769f-zlsf6" Sep 10 14:31:06.446993 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.446782 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d445769f-zlsf6"] Sep 10 14:31:06.449535 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:31:06.449508 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod369aaedc_43e4_4994_9bfe_7fd9e95b1f69.slice/crio-68726d0c5d7ca17db04acb2f5ad70f70702888d500b2c5803c8685b30a8e96e7 WatchSource:0}: Error finding container 68726d0c5d7ca17db04acb2f5ad70f70702888d500b2c5803c8685b30a8e96e7: Status 404 returned error can't find the container with id 68726d0c5d7ca17db04acb2f5ad70f70702888d500b2c5803c8685b30a8e96e7 Sep 10 14:31:06.773257 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:06.773181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d445769f-zlsf6" event={"ID":"369aaedc-43e4-4994-9bfe-7fd9e95b1f69","Type":"ContainerStarted","Data":"68726d0c5d7ca17db04acb2f5ad70f70702888d500b2c5803c8685b30a8e96e7"} Sep 10 14:31:07.780765 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:07.780730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d445769f-zlsf6" event={"ID":"369aaedc-43e4-4994-9bfe-7fd9e95b1f69","Type":"ContainerStarted","Data":"5e8db3f7b99e84b5b1cd2026cf4ee0b55229e99f40ab79c8135d8527500f8110"} Sep 10 14:31:07.795694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:07.795647 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d445769f-zlsf6" podStartSLOduration=1.6227427890000001 podStartE2EDuration="2.795630842s" totalImagesPullingTime="1.172888053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:07.794453205 +0000 UTC m=+685.648361881" watchObservedRunningTime="2026-09-10 14:31:07.795630842 +0000 UTC m=+685.649539522" Sep 10 14:31:08.394595 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:08.394544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:08.497915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:08.497880 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:09.416386 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:09.416352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:09.512337 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:09.512289 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:24.779094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:24.779059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:24.915184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:24.915136 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:25.203623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:25.203589 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:25.757588 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:25.757554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:26.285054 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:26.285019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:26.434043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:26.434005 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:26.753453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:26.753419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:28.637208 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:28.637172 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:28.715241 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:28.715207 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:31:28.717139 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:28.717100 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:29.824076 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:29.824041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:29.919422 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:29.919377 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:30.028670 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:30.028639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:30.203468 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:30.203432 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:30.305223 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:30.305186 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:32.862753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:32.862719 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:32.938077 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:32.938017 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:33.084977 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.084942 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:33.360109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.360075 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr"] Sep 10 14:31:33.364730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.364706 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.368218 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.368197 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ad4f3b47-openshift-default-dockercfg-75hxm\"" Sep 10 14:31:33.376404 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.376374 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr"] Sep 10 14:31:33.394088 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.394063 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:33.477914 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.477881 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-credential-socket\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.477949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-certs\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.477986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-socket\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.478039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-envoy\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.478062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-data\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.478082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-token\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478331 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.478117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb061a13-373a-4573-8bdd-6177c5ae6861-istiod-ca-cert\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478331 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.478168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-podinfo\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.478331 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.478191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm8dk\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-kube-api-access-nm8dk\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.568631 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.568594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:33.578625 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578597 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-certs\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.578768 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-socket\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.578768 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-envoy\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.578768 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-data\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.578768 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-token\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb061a13-373a-4573-8bdd-6177c5ae6861-istiod-ca-cert\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-podinfo\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm8dk\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-kube-api-access-nm8dk\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.578895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-credential-socket\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.579219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-certs\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.579256 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-data\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.579293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-socket\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.579383 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-credential-socket\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.579706 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.579683 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb061a13-373a-4573-8bdd-6177c5ae6861-istiod-ca-cert\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.581160 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.581143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-envoy\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.581369 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.581350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-podinfo\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.589323 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.589297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm8dk\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-kube-api-access-nm8dk\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.589409 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.589394 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-token\") pod \"e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.676629 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.676601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:33.950332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.950309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr"] Sep 10 14:31:33.952172 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:31:33.952146 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb061a13_373a_4573_8bdd_6177c5ae6861.slice/crio-08fae17b4a304cd7ae9da659c207ff1a7b3e46c1991ab44e2550c7a65b4bf034 WatchSource:0}: Error finding container 08fae17b4a304cd7ae9da659c207ff1a7b3e46c1991ab44e2550c7a65b4bf034: Status 404 returned error can't find the container with id 08fae17b4a304cd7ae9da659c207ff1a7b3e46c1991ab44e2550c7a65b4bf034 Sep 10 14:31:33.954410 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.954375 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:33.954490 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.954442 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:33.954490 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:33.954483 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:34.052155 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.052126 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:34.305864 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.305778 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq"] Sep 10 14:31:34.309376 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.309353 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.312302 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.312077 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-d24d4c-openshift-default-dockercfg-frpwl\"" Sep 10 14:31:34.319129 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.319074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq"] Sep 10 14:31:34.341577 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.341548 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:34.384869 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.384804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfk5\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-kube-api-access-7nfk5\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.384912 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-envoy\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.384943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-data\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.384976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-certs\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.385102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-token\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.385146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-podinfo\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.385171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-credential-socket\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.385197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istiod-ca-cert\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.385332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.385224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-socket\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.440233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.440200 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:34.485826 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfk5\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-kube-api-access-7nfk5\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.485994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-envoy\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.485994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-data\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.485994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-certs\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.485994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-token\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.485994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-podinfo\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.485994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.485986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-credential-socket\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486274 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istiod-ca-cert\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486274 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-socket\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486595 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486456 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-data\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486595 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486515 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-certs\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486595 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-socket\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486835 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486670 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-credential-socket\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.486835 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.486816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istiod-ca-cert\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.488760 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.488741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-envoy\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.488951 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.488928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-podinfo\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.494328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.494308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-token\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.494504 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.494486 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfk5\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-kube-api-access-7nfk5\") pod \"e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.621754 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.621685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:34.906942 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.906905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" event={"ID":"fb061a13-373a-4573-8bdd-6177c5ae6861","Type":"ContainerStarted","Data":"366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03"} Sep 10 14:31:34.907081 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.906958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" event={"ID":"fb061a13-373a-4573-8bdd-6177c5ae6861","Type":"ContainerStarted","Data":"08fae17b4a304cd7ae9da659c207ff1a7b3e46c1991ab44e2550c7a65b4bf034"} Sep 10 14:31:34.923152 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.922885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq"] Sep 10 14:31:34.925738 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:31:34.925709 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7f1a1e_36d9_4249_9a18_ea12b16da199.slice/crio-e360fa0f9aa532bdbfe7757b65128c1b784619a0d685b7235a47a9d66fdcf467 WatchSource:0}: Error finding container e360fa0f9aa532bdbfe7757b65128c1b784619a0d685b7235a47a9d66fdcf467: Status 404 returned error can't find the container with id e360fa0f9aa532bdbfe7757b65128c1b784619a0d685b7235a47a9d66fdcf467 Sep 10 14:31:34.928364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.928325 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:34.928468 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.928412 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:34.928468 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.928451 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:34.936141 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:34.936100 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podStartSLOduration=1.936085149 podStartE2EDuration="1.936085149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:34.935420779 +0000 UTC m=+712.789329471" watchObservedRunningTime="2026-09-10 14:31:34.936085149 +0000 UTC m=+712.789993819" Sep 10 14:31:35.677626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.677587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:35.682760 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.682734 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:35.914388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.914346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" event={"ID":"5c7f1a1e-36d9-4249-9a18-ea12b16da199","Type":"ContainerStarted","Data":"8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4"} Sep 10 14:31:35.914619 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.914413 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" event={"ID":"5c7f1a1e-36d9-4249-9a18-ea12b16da199","Type":"ContainerStarted","Data":"e360fa0f9aa532bdbfe7757b65128c1b784619a0d685b7235a47a9d66fdcf467"} Sep 10 14:31:35.914719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.914704 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:35.915882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.915859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:35.935514 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:35.935434 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podStartSLOduration=1.935421613 podStartE2EDuration="1.935421613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:35.933521389 +0000 UTC m=+713.787430067" watchObservedRunningTime="2026-09-10 14:31:35.935421613 +0000 UTC m=+713.789330290" Sep 10 14:31:36.386991 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:36.386951 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:36.470502 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:36.470463 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:36.622510 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:36.622478 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:36.627312 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:36.627288 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:36.919130 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:36.919100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:36.919921 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:36.919901 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:37.697206 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:37.697170 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:37.886461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:37.886425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:37.974681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:37.974600 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:38.354581 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:38.354493 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:38.456957 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:38.456892 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:39.529352 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:39.529303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:39.730904 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:39.730864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:39.980268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:39.980226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:40.193426 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:40.193391 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:40.292992 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:40.292907 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:42.589690 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:42.589657 2576 scope.go:117] "RemoveContainer" containerID="c52812e9761eea0b1367e885692babb2a84e4b73c4ce44c9dc65b9cea0cdf035" Sep 10 14:31:43.518315 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:43.518285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:31:43.777204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:43.777128 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:43.932257 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:43.932222 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:44.035602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:44.035514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:44.135713 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:44.135675 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:46.837877 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:46.837839 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:48.106812 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:48.106359 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:48.202607 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:48.202567 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:48.335191 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:48.335149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:50.052637 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:50.052603 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:50.623253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:50.623178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:51.956399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:51.955981 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 14:31:51.956399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:51.956069 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 14:31:52.960954 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:52.960910 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 14:31:52.961435 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:52.960986 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 14:31:53.442254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:53.442216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:53.806699 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:53.806659 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:53.999053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:53.998981 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl"] Sep 10 14:31:54.010143 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.010082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.012945 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.012922 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-12068cef-kserve-self-signed-certs\"" Sep 10 14:31:54.013562 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.013541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl"] Sep 10 14:31:54.023922 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.023896 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerID="366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03" exitCode=0 Sep 10 14:31:54.024032 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.023954 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" event={"ID":"fb061a13-373a-4573-8bdd-6177c5ae6861","Type":"ContainerDied","Data":"366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03"} Sep 10 14:31:54.024304 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.024290 2576 scope.go:117] "RemoveContainer" containerID="366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03" Sep 10 14:31:54.025466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.025423 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:54.025563 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.025493 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:54.025563 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.025524 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:54.080994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.080957 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w"] Sep 10 14:31:54.083365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.083337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.086128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.086107 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8ee208-openshift-default-dockercfg-bldb6\"" Sep 10 14:31:54.096239 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.096219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w"] Sep 10 14:31:54.111054 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.111030 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:54.172701 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.172655 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzchk\" (UniqueName: \"kubernetes.io/projected/c6977d85-e827-4442-a85b-056900d690d6-kube-api-access-hzchk\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.172912 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.172738 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-model-cache\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.172912 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.172782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6977d85-e827-4442-a85b-056900d690d6-tls-certs\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.172912 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.172887 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-home\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.173089 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.173003 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-dshm\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.173089 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.173032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-tmp-dir\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.264354 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.264280 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:54.273965 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.273939 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-podinfo\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274064 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.273973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7htk\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-kube-api-access-b7htk\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-dshm\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-data\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274230 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-tmp-dir\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274248 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istiod-ca-cert\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274315 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274295 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzchk\" (UniqueName: \"kubernetes.io/projected/c6977d85-e827-4442-a85b-056900d690d6-kube-api-access-hzchk\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-certs\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274433 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-model-cache\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274487 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6977d85-e827-4442-a85b-056900d690d6-tls-certs\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274568 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-socket\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274631 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-tmp-dir\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274631 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-credential-socket\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-home\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.274738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-envoy\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-token\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.274738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.274701 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-model-cache\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.275040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.275012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-home\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.276313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.276284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-dshm\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.277072 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.277053 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6977d85-e827-4442-a85b-056900d690d6-tls-certs\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.284637 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.284616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzchk\" (UniqueName: \"kubernetes.io/projected/c6977d85-e827-4442-a85b-056900d690d6-kube-api-access-hzchk\") pod \"e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.323021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.322991 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-certs\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377421 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-socket\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-credential-socket\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-envoy\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-token\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-podinfo\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7htk\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-kube-api-access-b7htk\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-data\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.377895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.377708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istiod-ca-cert\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.379303 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.378474 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-socket\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.379303 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.378642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istiod-ca-cert\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.379303 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.378772 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-certs\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.379567 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.379252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-data\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.379662 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.379443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-credential-socket\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.381954 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.381866 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-envoy\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.383137 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.383115 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-podinfo\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.390169 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.390121 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-token\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.391026 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.390980 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7htk\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-kube-api-access-b7htk\") pod \"e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.397877 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.397704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:54.429417 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.427341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:54.598502 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.598475 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl"] Sep 10 14:31:54.600336 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:31:54.600300 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6977d85_e827_4442_a85b_056900d690d6.slice/crio-054e336b79775111284c85e9040e3984652819afe3eec759a1e75845bc9cb4b4 WatchSource:0}: Error finding container 054e336b79775111284c85e9040e3984652819afe3eec759a1e75845bc9cb4b4: Status 404 returned error can't find the container with id 054e336b79775111284c85e9040e3984652819afe3eec759a1e75845bc9cb4b4 Sep 10 14:31:54.695286 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.695260 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w"] Sep 10 14:31:54.696350 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:31:54.696328 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a17e847_ed26_4b0d_8c6e_35197ce5b683.slice/crio-c9ce1336f37c0770b5d90cf76d978c4231b234d459bad95b03b0eababe36eca8 WatchSource:0}: Error finding container c9ce1336f37c0770b5d90cf76d978c4231b234d459bad95b03b0eababe36eca8: Status 404 returned error can't find the container with id c9ce1336f37c0770b5d90cf76d978c4231b234d459bad95b03b0eababe36eca8 Sep 10 14:31:54.698558 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.698524 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:54.698648 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.698590 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:54.698648 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:54.698625 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:55.030173 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.030138 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerID="8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4" exitCode=0 Sep 10 14:31:55.030571 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.030211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" event={"ID":"5c7f1a1e-36d9-4249-9a18-ea12b16da199","Type":"ContainerDied","Data":"8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4"} Sep 10 14:31:55.030757 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.030739 2576 scope.go:117] "RemoveContainer" containerID="8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4" Sep 10 14:31:55.031837 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.031814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" event={"ID":"1a17e847-ed26-4b0d-8c6e-35197ce5b683","Type":"ContainerStarted","Data":"864353a248ff63c688d36ac3719527ec8284265d447095c947e6ba41b8b86532"} Sep 10 14:31:55.031969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.031859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" event={"ID":"1a17e847-ed26-4b0d-8c6e-35197ce5b683","Type":"ContainerStarted","Data":"c9ce1336f37c0770b5d90cf76d978c4231b234d459bad95b03b0eababe36eca8"} Sep 10 14:31:55.032039 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.031990 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:55.032095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.032057 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:55.032143 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.032097 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:31:55.033836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.033813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" event={"ID":"fb061a13-373a-4573-8bdd-6177c5ae6861","Type":"ContainerStarted","Data":"a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385"} Sep 10 14:31:55.035383 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.035363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" event={"ID":"c6977d85-e827-4442-a85b-056900d690d6","Type":"ContainerStarted","Data":"97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4"} Sep 10 14:31:55.035456 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.035394 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" event={"ID":"c6977d85-e827-4442-a85b-056900d690d6","Type":"ContainerStarted","Data":"054e336b79775111284c85e9040e3984652819afe3eec759a1e75845bc9cb4b4"} Sep 10 14:31:55.300308 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.300200 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podStartSLOduration=1.3001817230000001 podStartE2EDuration="1.300181723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:55.253715493 +0000 UTC m=+733.107624170" watchObservedRunningTime="2026-09-10 14:31:55.300181723 +0000 UTC m=+733.154090399" Sep 10 14:31:55.398906 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.398851 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:55.410244 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.410208 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:55.436876 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.436823 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" podStartSLOduration=2.436773737 podStartE2EDuration="2.436773737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:31:55.301229599 +0000 UTC m=+733.155138275" watchObservedRunningTime="2026-09-10 14:31:55.436773737 +0000 UTC m=+733.290682413" Sep 10 14:31:55.575585 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.575492 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:55.677583 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.677546 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:31:55.678093 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.678058 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 14:31:55.678227 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.678132 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 14:31:55.765922 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.765883 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:55.863903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.863784 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:55.973705 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:55.973651 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:56.043181 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.043138 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" event={"ID":"5c7f1a1e-36d9-4249-9a18-ea12b16da199","Type":"ContainerStarted","Data":"635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae"} Sep 10 14:31:56.043588 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.043422 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:56.044388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.044361 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:31:56.621951 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.621921 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:31:56.622157 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.622135 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 14:31:56.622204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.622179 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 14:31:56.677465 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.677426 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 14:31:56.677678 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:56.677498 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 14:31:57.314506 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:57.314447 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:57.412941 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:57.412901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:57.622569 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:57.622484 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 14:31:57.622569 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:57.622533 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 14:31:57.677578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:57.677534 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 14:31:57.677773 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:57.677608 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 14:31:58.612812 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:58.612232 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:58.623395 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:58.623358 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 14:31:58.623526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:58.623422 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 14:31:58.682817 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:58.680209 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 14:31:58.682817 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:58.680277 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 14:31:58.709818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:58.708769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:59.036418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.036384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:59.208919 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.207752 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:59.300872 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.300219 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:31:59.623144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.623059 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 14:31:59.623144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.623121 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 14:31:59.677043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.677007 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 14:31:59.677324 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.677294 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 14:31:59.728911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:31:59.728882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:00.523709 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:00.523684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:00.623184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:00.623147 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Sep 10 14:32:00.623536 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:00.623207 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Sep 10 14:32:00.677600 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:00.677174 2576 patch_prober.go:28] interesting pod/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Sep 10 14:32:00.677600 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:00.677239 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Sep 10 14:32:01.307802 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.307754 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:01.625818 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.625716 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:32:01.626206 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.625989 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:32:01.626704 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.626680 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:32:01.681290 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.681259 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:32:01.681682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.681661 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:32:01.682439 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.682416 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:32:01.853693 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.853646 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q"] Sep 10 14:32:01.856333 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.856310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:01.858834 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.858813 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:32:01.858923 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.858818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:32:01.864435 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.864412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q"] Sep 10 14:32:01.958310 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:01.958256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q\" (UID: \"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:02.031783 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.031744 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:02.059548 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.059515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q\" (UID: \"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:02.060202 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.060184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q\" (UID: \"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:02.170018 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.169988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:02.445265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.445240 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q"] Sep 10 14:32:02.447477 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:32:02.447455 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b66f264_b1f6_4ec3_bc40_a4cb2578aff9.slice/crio-6687cc1b50a76f1d10c94046324f924be2efe9f97a2f0b874830bb6ddca827c6 WatchSource:0}: Error finding container 6687cc1b50a76f1d10c94046324f924be2efe9f97a2f0b874830bb6ddca827c6: Status 404 returned error can't find the container with id 6687cc1b50a76f1d10c94046324f924be2efe9f97a2f0b874830bb6ddca827c6 Sep 10 14:32:02.543526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.543425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:02.644142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.644105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:02.746189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.746145 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:02.843425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:02.843340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:03.085323 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:03.085284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" event={"ID":"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9","Type":"ContainerStarted","Data":"6687cc1b50a76f1d10c94046324f924be2efe9f97a2f0b874830bb6ddca827c6"} Sep 10 14:32:04.323509 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:04.323472 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:32:04.323940 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:04.323533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:32:04.337515 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:04.337484 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:32:05.107294 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:05.107260 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:32:06.105494 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:06.105458 2576 generic.go:356] "Generic (PLEG): container finished" podID="0b66f264-b1f6-4ec3-bc40-a4cb2578aff9" containerID="e7ff4203e3719e3e4067e35f2defea7e4edc1734f14b273c0b8b08b049c60646" exitCode=0 Sep 10 14:32:06.105871 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:06.105520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" event={"ID":"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9","Type":"ContainerDied","Data":"e7ff4203e3719e3e4067e35f2defea7e4edc1734f14b273c0b8b08b049c60646"} Sep 10 14:32:06.512502 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:06.512466 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:32:07.290541 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:07.290516 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:07.432767 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:07.432735 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9-maas-api-service-ca\") pod \"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9\" (UID: \"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9\") " Sep 10 14:32:07.433180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:07.433153 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0b66f264-b1f6-4ec3-bc40-a4cb2578aff9" (UID: "0b66f264-b1f6-4ec3-bc40-a4cb2578aff9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:32:07.534332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:07.534296 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:08.119093 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:08.119064 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" Sep 10 14:32:08.119265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:08.119108 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q" event={"ID":"0b66f264-b1f6-4ec3-bc40-a4cb2578aff9","Type":"ContainerDied","Data":"6687cc1b50a76f1d10c94046324f924be2efe9f97a2f0b874830bb6ddca827c6"} Sep 10 14:32:08.119265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:08.119151 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6687cc1b50a76f1d10c94046324f924be2efe9f97a2f0b874830bb6ddca827c6" Sep 10 14:32:08.421579 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:08.421549 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:08.523592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:08.523554 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:17.830054 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:17.830015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:18.420232 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.420154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:18.491515 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.491418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:18.590842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.590809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:18.599596 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.599568 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q"] Sep 10 14:32:18.606299 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.606275 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ad4f3b47-bkq4q"] Sep 10 14:32:18.758439 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.758344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:18.764022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.763994 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b66f264-b1f6-4ec3-bc40-a4cb2578aff9" path="/var/lib/kubelet/pods/0b66f264-b1f6-4ec3-bc40-a4cb2578aff9/volumes" Sep 10 14:32:18.764285 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.764270 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr"] Sep 10 14:32:18.764483 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.764462 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" containerID="cri-o://a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385" gracePeriod=30 Sep 10 14:32:18.832708 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:18.832672 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:19.175401 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.175365 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerID="864353a248ff63c688d36ac3719527ec8284265d447095c947e6ba41b8b86532" exitCode=0 Sep 10 14:32:19.175566 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.175432 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" event={"ID":"1a17e847-ed26-4b0d-8c6e-35197ce5b683","Type":"ContainerDied","Data":"864353a248ff63c688d36ac3719527ec8284265d447095c947e6ba41b8b86532"} Sep 10 14:32:19.175840 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.175823 2576 scope.go:117] "RemoveContainer" containerID="864353a248ff63c688d36ac3719527ec8284265d447095c947e6ba41b8b86532" Sep 10 14:32:19.177082 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.177051 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:32:19.177203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.177138 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:32:19.177203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.177188 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:32:19.235354 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.235331 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:19.357884 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.357846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:19.465613 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.465530 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:19.604690 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.604641 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:19.794417 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.794326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:19.936864 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:19.936831 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:20.185254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:20.185222 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" event={"ID":"1a17e847-ed26-4b0d-8c6e-35197ce5b683","Type":"ContainerStarted","Data":"1f265db57379eb8cf9cafe93348b302abc8476132ac8abaf1fc30b9df56fff76"} Sep 10 14:32:20.398582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:20.398540 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:32:20.398867 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:20.398842 2576 patch_prober.go:28] interesting pod/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 14:32:20.398923 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:20.398895 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 14:32:21.398852 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:21.398817 2576 patch_prober.go:28] interesting pod/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 14:32:21.399329 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:21.398893 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 14:32:22.285827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.285770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:22.397054 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.397019 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:22.398728 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.398699 2576 patch_prober.go:28] interesting pod/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 14:32:22.398868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.398762 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 14:32:22.504423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.504364 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:22.642626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.642594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:22.741708 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.741670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:22.838558 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:22.838522 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:23.123352 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:23.123261 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:23.300455 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:23.300417 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:23.398635 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:23.398592 2576 patch_prober.go:28] interesting pod/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 14:32:23.398847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:23.398661 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 14:32:24.084463 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.084435 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:32:24.108713 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108688 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-credential-socket\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.108898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108725 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb061a13-373a-4573-8bdd-6177c5ae6861-istiod-ca-cert\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.108898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108763 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-certs\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.108898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108783 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-data\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.108898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108856 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm8dk\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-kube-api-access-nm8dk\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.108898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108887 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-envoy\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.109182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108961 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-socket\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.109182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-podinfo\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.109182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.108992 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:32:24.109182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109017 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:32:24.109182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-token\") pod \"fb061a13-373a-4573-8bdd-6177c5ae6861\" (UID: \"fb061a13-373a-4573-8bdd-6177c5ae6861\") " Sep 10 14:32:24.109182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109131 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb061a13-373a-4573-8bdd-6177c5ae6861-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:32:24.109525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109365 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.109525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109384 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb061a13-373a-4573-8bdd-6177c5ae6861-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.109525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109397 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.109674 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109597 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:32:24.109733 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.109662 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-data" (OuterVolumeSpecName: "istio-data") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:32:24.111712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.111595 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:32:24.111865 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.111715 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:32:24.112364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.112338 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:32:24.112490 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.112470 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-kube-api-access-nm8dk" (OuterVolumeSpecName: "kube-api-access-nm8dk") pod "fb061a13-373a-4573-8bdd-6177c5ae6861" (UID: "fb061a13-373a-4573-8bdd-6177c5ae6861"). InnerVolumeSpecName "kube-api-access-nm8dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:32:24.206466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.206430 2576 generic.go:356] "Generic (PLEG): container finished" podID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerID="a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385" exitCode=0 Sep 10 14:32:24.206611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.206477 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" Sep 10 14:32:24.206611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.206514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" event={"ID":"fb061a13-373a-4573-8bdd-6177c5ae6861","Type":"ContainerDied","Data":"a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385"} Sep 10 14:32:24.206611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.206570 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr" event={"ID":"fb061a13-373a-4573-8bdd-6177c5ae6861","Type":"ContainerDied","Data":"08fae17b4a304cd7ae9da659c207ff1a7b3e46c1991ab44e2550c7a65b4bf034"} Sep 10 14:32:24.206611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.206600 2576 scope.go:117] "RemoveContainer" containerID="a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385" Sep 10 14:32:24.210527 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.210504 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.210606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.210533 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.210606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.210548 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.210606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.210561 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.210606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.210574 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb061a13-373a-4573-8bdd-6177c5ae6861-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.210606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.210588 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm8dk\" (UniqueName: \"kubernetes.io/projected/fb061a13-373a-4573-8bdd-6177c5ae6861-kube-api-access-nm8dk\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:24.215990 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.215973 2576 scope.go:117] "RemoveContainer" containerID="366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03" Sep 10 14:32:24.223679 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.223662 2576 scope.go:117] "RemoveContainer" containerID="a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385" Sep 10 14:32:24.223956 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:32:24.223938 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385\": container with ID starting with a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385 not found: ID does not exist" containerID="a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385" Sep 10 14:32:24.224014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.223963 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385"} err="failed to get container status \"a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385\": rpc error: code = NotFound desc = could not find container \"a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385\": container with ID starting with a69b49a9fcb1febe86a5a608f49ff0957813e43eb7a73d0dcc49b64e8fdeb385 not found: ID does not exist" Sep 10 14:32:24.224014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.223981 2576 scope.go:117] "RemoveContainer" containerID="366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03" Sep 10 14:32:24.224218 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:32:24.224200 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03\": container with ID starting with 366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03 not found: ID does not exist" containerID="366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03" Sep 10 14:32:24.224272 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.224228 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03"} err="failed to get container status \"366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03\": rpc error: code = NotFound desc = could not find container \"366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03\": container with ID starting with 366fc9ae8bfe17bf7dfee3d98b96869b007f1def935198917dd7fd15ff635f03 not found: ID does not exist" Sep 10 14:32:24.245496 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.245472 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr"] Sep 10 14:32:24.252714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.252694 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ad4f3b47-openshift-default-59b786598b-4l9fr"] Sep 10 14:32:24.398640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.398603 2576 patch_prober.go:28] interesting pod/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Sep 10 14:32:24.398837 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.398660 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Sep 10 14:32:24.432251 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.432223 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:32:24.783868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:24.783388 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" path="/var/lib/kubelet/pods/fb061a13-373a-4573-8bdd-6177c5ae6861/volumes" Sep 10 14:32:25.401932 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:25.401899 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:32:25.402326 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:25.402188 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:32:25.402888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:25.402872 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:32:31.270702 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:31.270667 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:31.401458 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:31.401419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:41.781805 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:41.781755 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:32:47.316078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316035 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85"] Sep 10 14:32:47.316459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316428 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" Sep 10 14:32:47.316459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316443 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" Sep 10 14:32:47.316459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316458 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" Sep 10 14:32:47.316564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316464 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" Sep 10 14:32:47.316564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316470 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b66f264-b1f6-4ec3-bc40-a4cb2578aff9" containerName="revoke-keys" Sep 10 14:32:47.316564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316476 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b66f264-b1f6-4ec3-bc40-a4cb2578aff9" containerName="revoke-keys" Sep 10 14:32:47.316564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316534 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" Sep 10 14:32:47.316564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316542 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b66f264-b1f6-4ec3-bc40-a4cb2578aff9" containerName="revoke-keys" Sep 10 14:32:47.316564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.316551 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb061a13-373a-4573-8bdd-6177c5ae6861" containerName="istio-proxy" Sep 10 14:32:47.319368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.319351 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:47.322067 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.322046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:32:47.322177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.322136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:32:47.327090 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.327064 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85"] Sep 10 14:32:47.410762 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.410735 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6253601e-0b5f-49f7-b804-85a2c132f8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85\" (UID: \"6253601e-0b5f-49f7-b804-85a2c132f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:47.511998 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.511964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6253601e-0b5f-49f7-b804-85a2c132f8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85\" (UID: \"6253601e-0b5f-49f7-b804-85a2c132f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:47.512616 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.512595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6253601e-0b5f-49f7-b804-85a2c132f8e6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85\" (UID: \"6253601e-0b5f-49f7-b804-85a2c132f8e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:47.630875 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.630845 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:47.901600 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:47.901574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85"] Sep 10 14:32:47.903472 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:32:47.903442 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6253601e_0b5f_49f7_b804_85a2c132f8e6.slice/crio-0baf09deaaa72ceb450abf7315a223f5d560ba5459ae99965cab661b32a63c67 WatchSource:0}: Error finding container 0baf09deaaa72ceb450abf7315a223f5d560ba5459ae99965cab661b32a63c67: Status 404 returned error can't find the container with id 0baf09deaaa72ceb450abf7315a223f5d560ba5459ae99965cab661b32a63c67 Sep 10 14:32:48.339799 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:48.339702 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" event={"ID":"6253601e-0b5f-49f7-b804-85a2c132f8e6","Type":"ContainerStarted","Data":"b9da05b761f9fc57f0dfd011ab0f115c19bcbf7a565ff9185004dfbc40efc405"} Sep 10 14:32:48.339799 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:48.339768 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" event={"ID":"6253601e-0b5f-49f7-b804-85a2c132f8e6","Type":"ContainerStarted","Data":"0baf09deaaa72ceb450abf7315a223f5d560ba5459ae99965cab661b32a63c67"} Sep 10 14:32:48.359352 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:48.359296 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" podStartSLOduration=1.3592816079999999 podStartE2EDuration="1.359281608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:48.357532256 +0000 UTC m=+786.211440932" watchObservedRunningTime="2026-09-10 14:32:48.359281608 +0000 UTC m=+786.213190284" Sep 10 14:32:49.346827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:49.346772 2576 generic.go:356] "Generic (PLEG): container finished" podID="6253601e-0b5f-49f7-b804-85a2c132f8e6" containerID="b9da05b761f9fc57f0dfd011ab0f115c19bcbf7a565ff9185004dfbc40efc405" exitCode=0 Sep 10 14:32:49.347189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:49.346862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" event={"ID":"6253601e-0b5f-49f7-b804-85a2c132f8e6","Type":"ContainerDied","Data":"b9da05b761f9fc57f0dfd011ab0f115c19bcbf7a565ff9185004dfbc40efc405"} Sep 10 14:32:50.490966 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:50.490942 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:50.640461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:50.640435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6253601e-0b5f-49f7-b804-85a2c132f8e6-maas-api-service-ca\") pod \"6253601e-0b5f-49f7-b804-85a2c132f8e6\" (UID: \"6253601e-0b5f-49f7-b804-85a2c132f8e6\") " Sep 10 14:32:50.640847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:50.640821 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6253601e-0b5f-49f7-b804-85a2c132f8e6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6253601e-0b5f-49f7-b804-85a2c132f8e6" (UID: "6253601e-0b5f-49f7-b804-85a2c132f8e6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:32:50.741926 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:50.741891 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6253601e-0b5f-49f7-b804-85a2c132f8e6-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:32:51.359387 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:51.359349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" event={"ID":"6253601e-0b5f-49f7-b804-85a2c132f8e6","Type":"ContainerDied","Data":"0baf09deaaa72ceb450abf7315a223f5d560ba5459ae99965cab661b32a63c67"} Sep 10 14:32:51.359387 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:51.359379 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85" Sep 10 14:32:51.359593 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:51.359406 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0baf09deaaa72ceb450abf7315a223f5d560ba5459ae99965cab661b32a63c67" Sep 10 14:32:53.959908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:53.959869 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:54.097244 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.097202 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:54.180483 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.180452 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85"] Sep 10 14:32:54.183590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.183566 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d320da1d-mgt85"] Sep 10 14:32:54.192446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.192418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:54.263139 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.263062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:54.341416 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.341353 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:54.760590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.760557 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6253601e-0b5f-49f7-b804-85a2c132f8e6" path="/var/lib/kubelet/pods/6253601e-0b5f-49f7-b804-85a2c132f8e6/volumes" Sep 10 14:32:54.794886 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:54.794855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:55.056000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.055913 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l"] Sep 10 14:32:55.056469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.056449 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6253601e-0b5f-49f7-b804-85a2c132f8e6" containerName="revoke-keys" Sep 10 14:32:55.056555 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.056472 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6253601e-0b5f-49f7-b804-85a2c132f8e6" containerName="revoke-keys" Sep 10 14:32:55.056615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.056604 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6253601e-0b5f-49f7-b804-85a2c132f8e6" containerName="revoke-keys" Sep 10 14:32:55.058898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.058868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.061757 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.061737 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-49ac64b7-openshift-default-dockercfg-zcrrh\"" Sep 10 14:32:55.072040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.072008 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l"] Sep 10 14:32:55.105390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.105360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:55.183553 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-credential-socket\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.183553 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-certs\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.183823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2r7\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-kube-api-access-lk2r7\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.183823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3baa2a14-71f8-40b0-b020-5a19839febf1-istiod-ca-cert\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.183823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183701 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-envoy\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.183823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-token\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.183823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183812 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-data\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.184084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-podinfo\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.184084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.183917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-socket\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-envoy\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-token\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285320 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-data\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-podinfo\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-socket\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-credential-socket\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-certs\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2r7\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-kube-api-access-lk2r7\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.285616 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3baa2a14-71f8-40b0-b020-5a19839febf1-istiod-ca-cert\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.286415 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3baa2a14-71f8-40b0-b020-5a19839febf1-istiod-ca-cert\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.289174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-envoy\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.290317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.289770 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-data\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.291070 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.290564 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-certs\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.291070 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.290608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-credential-socket\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.291070 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.290636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-socket\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.293615 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.293590 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-podinfo\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.300650 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.300627 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-token\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.301209 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.301192 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2r7\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-kube-api-access-lk2r7\") pod \"e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.370140 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.370055 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:55.377223 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.377190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:55.649526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.649502 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l"] Sep 10 14:32:55.651284 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:32:55.651255 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3baa2a14_71f8_40b0_b020_5a19839febf1.slice/crio-8006574b3a99bf09e9547bc57fd5b3e1b3364b6b794a86719b49ec070ec91db0 WatchSource:0}: Error finding container 8006574b3a99bf09e9547bc57fd5b3e1b3364b6b794a86719b49ec070ec91db0: Status 404 returned error can't find the container with id 8006574b3a99bf09e9547bc57fd5b3e1b3364b6b794a86719b49ec070ec91db0 Sep 10 14:32:55.653626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.653582 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:32:55.653745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.653670 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:32:55.653745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:55.653712 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:32:56.388019 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:56.387976 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" event={"ID":"3baa2a14-71f8-40b0-b020-5a19839febf1","Type":"ContainerStarted","Data":"906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da"} Sep 10 14:32:56.388442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:56.388042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" event={"ID":"3baa2a14-71f8-40b0-b020-5a19839febf1","Type":"ContainerStarted","Data":"8006574b3a99bf09e9547bc57fd5b3e1b3364b6b794a86719b49ec070ec91db0"} Sep 10 14:32:56.408783 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:56.408617 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podStartSLOduration=1.408598897 podStartE2EDuration="1.408598897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:32:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:32:56.407883346 +0000 UTC m=+794.261792037" watchObservedRunningTime="2026-09-10 14:32:56.408598897 +0000 UTC m=+794.262507575" Sep 10 14:32:57.371250 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:57.371218 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:57.376348 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:57.376318 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:57.395672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:57.395602 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:57.397156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:57.397133 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:32:57.834580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:57.834532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:57.930330 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:57.930295 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:58.977570 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:58.977536 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:59.075119 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:59.075084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:59.178181 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:59.178148 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:59.444654 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:59.444619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:59.532288 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:59.532252 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:32:59.962453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:32:59.962420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:00.063662 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:00.063628 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:00.407106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:00.407065 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:33:18.105803 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:18.105745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:33:19.519773 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:19.519735 2576 generic.go:356] "Generic (PLEG): container finished" podID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerID="906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da" exitCode=0 Sep 10 14:33:19.520194 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:19.519826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" event={"ID":"3baa2a14-71f8-40b0-b020-5a19839febf1","Type":"ContainerDied","Data":"906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da"} Sep 10 14:33:19.520263 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:19.520248 2576 scope.go:117] "RemoveContainer" containerID="906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da" Sep 10 14:33:19.521417 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:19.521389 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:33:19.521482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:19.521458 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:33:19.521517 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:19.521491 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:33:20.526999 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:20.526968 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" event={"ID":"3baa2a14-71f8-40b0-b020-5a19839febf1","Type":"ContainerStarted","Data":"856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a"} Sep 10 14:33:21.371065 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:21.370966 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:33:21.371413 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:21.371391 2576 patch_prober.go:28] interesting pod/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 14:33:21.371575 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:21.371547 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 14:33:21.758398 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:21.758368 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:22.370909 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:22.370417 2576 patch_prober.go:28] interesting pod/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 14:33:22.370909 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:22.370483 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 14:33:23.371074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.371028 2576 patch_prober.go:28] interesting pod/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 14:33:23.371585 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.371141 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 14:33:23.523692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.523648 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4"] Sep 10 14:33:23.526695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.526674 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:23.529244 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.529228 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:33:23.529439 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.529262 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:33:23.534406 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.534383 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4"] Sep 10 14:33:23.552592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.552559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f94cf7-a0ed-483e-9757-6be1b882e646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4\" (UID: \"13f94cf7-a0ed-483e-9757-6be1b882e646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:23.654024 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.653985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f94cf7-a0ed-483e-9757-6be1b882e646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4\" (UID: \"13f94cf7-a0ed-483e-9757-6be1b882e646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:23.654777 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.654748 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f94cf7-a0ed-483e-9757-6be1b882e646-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4\" (UID: \"13f94cf7-a0ed-483e-9757-6be1b882e646\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:23.838328 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:23.838292 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:24.130868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:24.130835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4"] Sep 10 14:33:24.134187 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:33:24.134157 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f94cf7_a0ed_483e_9757_6be1b882e646.slice/crio-c498164d53e8b5904557be87a4e9f8d29a0d57b72a949a6f514fecbe1f742d08 WatchSource:0}: Error finding container c498164d53e8b5904557be87a4e9f8d29a0d57b72a949a6f514fecbe1f742d08: Status 404 returned error can't find the container with id c498164d53e8b5904557be87a4e9f8d29a0d57b72a949a6f514fecbe1f742d08 Sep 10 14:33:24.371003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:24.370820 2576 patch_prober.go:28] interesting pod/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 14:33:24.371003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:24.370882 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 14:33:24.550786 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:24.550709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" event={"ID":"13f94cf7-a0ed-483e-9757-6be1b882e646","Type":"ContainerStarted","Data":"a7ed09907df3dcfa5b9ea124e9616da099ecb4c83e56342fef938cbcfa17c3fb"} Sep 10 14:33:24.550955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:24.550784 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" event={"ID":"13f94cf7-a0ed-483e-9757-6be1b882e646","Type":"ContainerStarted","Data":"c498164d53e8b5904557be87a4e9f8d29a0d57b72a949a6f514fecbe1f742d08"} Sep 10 14:33:25.370872 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:25.370835 2576 patch_prober.go:28] interesting pod/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Sep 10 14:33:25.371203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:25.371172 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Sep 10 14:33:25.560763 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:25.560675 2576 generic.go:356] "Generic (PLEG): container finished" podID="13f94cf7-a0ed-483e-9757-6be1b882e646" containerID="a7ed09907df3dcfa5b9ea124e9616da099ecb4c83e56342fef938cbcfa17c3fb" exitCode=0 Sep 10 14:33:25.561104 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:25.561080 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" event={"ID":"13f94cf7-a0ed-483e-9757-6be1b882e646","Type":"ContainerDied","Data":"a7ed09907df3dcfa5b9ea124e9616da099ecb4c83e56342fef938cbcfa17c3fb"} Sep 10 14:33:26.375611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.375579 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:33:26.376052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.375975 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:33:26.376629 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.376608 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:33:26.698651 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.698629 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:26.800286 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.800257 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f94cf7-a0ed-483e-9757-6be1b882e646-maas-api-service-ca\") pod \"13f94cf7-a0ed-483e-9757-6be1b882e646\" (UID: \"13f94cf7-a0ed-483e-9757-6be1b882e646\") " Sep 10 14:33:26.800585 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.800564 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f94cf7-a0ed-483e-9757-6be1b882e646-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "13f94cf7-a0ed-483e-9757-6be1b882e646" (UID: "13f94cf7-a0ed-483e-9757-6be1b882e646"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:33:26.901442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:26.901416 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/13f94cf7-a0ed-483e-9757-6be1b882e646-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:27.572880 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:27.572847 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" Sep 10 14:33:27.573319 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:27.572890 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4" event={"ID":"13f94cf7-a0ed-483e-9757-6be1b882e646","Type":"ContainerDied","Data":"c498164d53e8b5904557be87a4e9f8d29a0d57b72a949a6f514fecbe1f742d08"} Sep 10 14:33:27.573319 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:27.572936 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c498164d53e8b5904557be87a4e9f8d29a0d57b72a949a6f514fecbe1f742d08" Sep 10 14:33:29.462159 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:29.462115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:29.563849 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:29.563808 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:29.635242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:29.635212 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4"] Sep 10 14:33:29.637329 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:29.637307 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49ac64b7-njmb4"] Sep 10 14:33:29.682134 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:29.682096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:29.783058 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:29.782968 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:30.765139 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:30.765099 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f94cf7-a0ed-483e-9757-6be1b882e646" path="/var/lib/kubelet/pods/13f94cf7-a0ed-483e-9757-6be1b882e646/volumes" Sep 10 14:33:30.781828 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:30.781772 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l"] Sep 10 14:33:30.782004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:30.781982 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" containerID="cri-o://856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a" gracePeriod=30 Sep 10 14:33:30.808891 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:30.808864 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:31.249480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:31.249446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:31.538775 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:31.538666 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:31.887368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:31.887336 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:33.696630 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:33.696591 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:33.786600 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:33.786565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:34.630907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:34.630874 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:34.830011 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:34.829977 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:35.837065 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:35.837034 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:35.936995 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:35.936964 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:36.037945 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.037923 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:33:36.187572 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187539 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-certs\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.187572 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187574 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-token\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.187778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187616 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk2r7\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-kube-api-access-lk2r7\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.187778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3baa2a14-71f8-40b0-b020-5a19839febf1-istiod-ca-cert\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.187778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187673 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-data\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.187778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187707 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-credential-socket\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.188080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-envoy\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.188080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187834 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-socket\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.188080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187876 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:33:36.188080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.187884 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-podinfo\") pod \"3baa2a14-71f8-40b0-b020-5a19839febf1\" (UID: \"3baa2a14-71f8-40b0-b020-5a19839febf1\") " Sep 10 14:33:36.188080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188037 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-data" (OuterVolumeSpecName: "istio-data") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:33:36.188080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188053 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:33:36.188390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188129 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3baa2a14-71f8-40b0-b020-5a19839febf1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:33:36.188390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188255 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:33:36.188390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188382 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.188543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188412 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3baa2a14-71f8-40b0-b020-5a19839febf1-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.188543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188426 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.188543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188438 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.188543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.188453 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.189757 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.189734 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-token" (OuterVolumeSpecName: "istio-token") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:33:36.189902 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.189846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:33:36.190123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.190109 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-kube-api-access-lk2r7" (OuterVolumeSpecName: "kube-api-access-lk2r7") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "kube-api-access-lk2r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:33:36.190260 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.190241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3baa2a14-71f8-40b0-b020-5a19839febf1" (UID: "3baa2a14-71f8-40b0-b020-5a19839febf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:33:36.289869 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.289839 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.289869 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.289864 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.289869 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.289873 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.290063 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.289881 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2r7\" (UniqueName: \"kubernetes.io/projected/3baa2a14-71f8-40b0-b020-5a19839febf1-kube-api-access-lk2r7\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:33:36.619009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.618924 2576 generic.go:356] "Generic (PLEG): container finished" podID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerID="856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a" exitCode=0 Sep 10 14:33:36.619009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.618970 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" Sep 10 14:33:36.619009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.618983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" event={"ID":"3baa2a14-71f8-40b0-b020-5a19839febf1","Type":"ContainerDied","Data":"856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a"} Sep 10 14:33:36.619258 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.619025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l" event={"ID":"3baa2a14-71f8-40b0-b020-5a19839febf1","Type":"ContainerDied","Data":"8006574b3a99bf09e9547bc57fd5b3e1b3364b6b794a86719b49ec070ec91db0"} Sep 10 14:33:36.619258 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.619046 2576 scope.go:117] "RemoveContainer" containerID="856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a" Sep 10 14:33:36.628121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.628105 2576 scope.go:117] "RemoveContainer" containerID="906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da" Sep 10 14:33:36.636112 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.636095 2576 scope.go:117] "RemoveContainer" containerID="856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a" Sep 10 14:33:36.636351 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:33:36.636331 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a\": container with ID starting with 856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a not found: ID does not exist" containerID="856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a" Sep 10 14:33:36.636388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.636360 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a"} err="failed to get container status \"856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a\": rpc error: code = NotFound desc = could not find container \"856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a\": container with ID starting with 856d52fe9480acb94ede88d28c28230b05d6cd9f96a39c837771190cf57c9d3a not found: ID does not exist" Sep 10 14:33:36.636388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.636382 2576 scope.go:117] "RemoveContainer" containerID="906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da" Sep 10 14:33:36.636655 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:33:36.636637 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da\": container with ID starting with 906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da not found: ID does not exist" containerID="906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da" Sep 10 14:33:36.636692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.636661 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da"} err="failed to get container status \"906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da\": rpc error: code = NotFound desc = could not find container \"906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da\": container with ID starting with 906a7313027ff12754174287c3fc4b2fb516d77306255b88fdf42f419cc025da not found: ID does not exist" Sep 10 14:33:36.641691 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.641672 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l"] Sep 10 14:33:36.645500 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.645481 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-49ac64b7-openshift-default-7976bbd5fb-bh77l"] Sep 10 14:33:36.715942 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.715910 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:33:36.760859 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:36.760822 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" path="/var/lib/kubelet/pods/3baa2a14-71f8-40b0-b020-5a19839febf1/volumes" Sep 10 14:33:47.525160 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:47.525121 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:53.810024 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.809989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:33:53.898297 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898267 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp"] Sep 10 14:33:53.898650 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898636 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" Sep 10 14:33:53.898714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898652 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" Sep 10 14:33:53.898714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898676 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f94cf7-a0ed-483e-9757-6be1b882e646" containerName="revoke-keys" Sep 10 14:33:53.898714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898683 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f94cf7-a0ed-483e-9757-6be1b882e646" containerName="revoke-keys" Sep 10 14:33:53.898836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898741 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" Sep 10 14:33:53.898836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898749 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" Sep 10 14:33:53.898836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.898757 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f94cf7-a0ed-483e-9757-6be1b882e646" containerName="revoke-keys" Sep 10 14:33:53.901334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.901319 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:53.905530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.905509 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-600197\"/\"kube-root-ca.crt\"" Sep 10 14:33:53.905530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.905519 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-600197\"/\"openshift-service-ca.crt\"" Sep 10 14:33:53.905530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.905510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-600197\"/\"test-model-600197-kserve-self-signed-certs\"" Sep 10 14:33:53.905805 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.905510 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-600197\"/\"default-dockercfg-x2pcm\"" Sep 10 14:33:53.911403 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.911380 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp"] Sep 10 14:33:53.946303 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.946278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-home\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:53.946453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.946346 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-dshm\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:53.946453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.946433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72405153-6a4b-46a9-91e5-27f9a3255a83-tls-certs\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:53.946594 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.946458 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-tmp-dir\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:53.946666 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.946603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-model-cache\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:53.946666 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:53.946633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t446f\" (UniqueName: \"kubernetes.io/projected/72405153-6a4b-46a9-91e5-27f9a3255a83-kube-api-access-t446f\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.015001 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.014972 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:54.047326 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-dshm\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72405153-6a4b-46a9-91e5-27f9a3255a83-tls-certs\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-tmp-dir\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047416 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-model-cache\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t446f\" (UniqueName: \"kubernetes.io/projected/72405153-6a4b-46a9-91e5-27f9a3255a83-kube-api-access-t446f\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047653 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-home\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047816 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-model-cache\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047925 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047867 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-home\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.047925 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.047871 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-tmp-dir\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.049558 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.049536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-dshm\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.049787 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.049771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72405153-6a4b-46a9-91e5-27f9a3255a83-tls-certs\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.056051 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.056024 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t446f\" (UniqueName: \"kubernetes.io/projected/72405153-6a4b-46a9-91e5-27f9a3255a83-kube-api-access-t446f\") pod \"test-model-600197-kserve-586597d96c-9wxpp\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.210761 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.210724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:33:54.482606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.482560 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp"] Sep 10 14:33:54.709147 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.709109 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" event={"ID":"72405153-6a4b-46a9-91e5-27f9a3255a83","Type":"ContainerStarted","Data":"daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3"} Sep 10 14:33:54.709309 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.709158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" event={"ID":"72405153-6a4b-46a9-91e5-27f9a3255a83","Type":"ContainerStarted","Data":"f816ae1d5d15e3e045e330417ba671cc26a76b5c2e7b24194a698dc7d1b6071d"} Sep 10 14:33:54.729367 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:54.729313 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" podStartSLOduration=1.729296963 podStartE2EDuration="1.729296963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:54.728180105 +0000 UTC m=+852.582088783" watchObservedRunningTime="2026-09-10 14:33:54.729296963 +0000 UTC m=+852.583205686" Sep 10 14:33:56.333300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:56.333269 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl"] Sep 10 14:33:56.352750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:56.352717 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:56.818965 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:56.818926 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:57.554293 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.554251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:57.638417 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.638382 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d"] Sep 10 14:33:57.638783 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.638770 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" Sep 10 14:33:57.638856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.638785 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3baa2a14-71f8-40b0-b020-5a19839febf1" containerName="istio-proxy" Sep 10 14:33:57.641209 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.641194 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:33:57.644974 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.644954 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:33:57.645514 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.645498 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:33:57.660004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.659975 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d"] Sep 10 14:33:57.676081 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.676051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6ef13-b379-4d77-94cb-3941c90a92ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d24d4c-hm69d\" (UID: \"0bd6ef13-b379-4d77-94cb-3941c90a92ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:33:57.777079 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.777042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6ef13-b379-4d77-94cb-3941c90a92ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d24d4c-hm69d\" (UID: \"0bd6ef13-b379-4d77-94cb-3941c90a92ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:33:57.777743 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.777717 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6ef13-b379-4d77-94cb-3941c90a92ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-d24d4c-hm69d\" (UID: \"0bd6ef13-b379-4d77-94cb-3941c90a92ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:33:57.951448 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.951418 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:33:57.961460 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:57.961423 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:58.228010 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:58.227985 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d"] Sep 10 14:33:58.230373 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:33:58.230345 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd6ef13_b379_4d77_94cb_3941c90a92ff.slice/crio-a8a874c14b94f55da7f1695777e84dbcd9e14190899884bbaf2db07e9ed6940c WatchSource:0}: Error finding container a8a874c14b94f55da7f1695777e84dbcd9e14190899884bbaf2db07e9ed6940c: Status 404 returned error can't find the container with id a8a874c14b94f55da7f1695777e84dbcd9e14190899884bbaf2db07e9ed6940c Sep 10 14:33:58.729886 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:58.729847 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" event={"ID":"0bd6ef13-b379-4d77-94cb-3941c90a92ff","Type":"ContainerStarted","Data":"cd190a6327c3620ba8ee67362f5a073260c1df1caba968d30651f8577a12b3f6"} Sep 10 14:33:58.730248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:58.729909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" event={"ID":"0bd6ef13-b379-4d77-94cb-3941c90a92ff","Type":"ContainerStarted","Data":"a8a874c14b94f55da7f1695777e84dbcd9e14190899884bbaf2db07e9ed6940c"} Sep 10 14:33:58.751903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:58.751855 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" podStartSLOduration=1.751841228 podStartE2EDuration="1.751841228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:33:58.749610992 +0000 UTC m=+856.603519667" watchObservedRunningTime="2026-09-10 14:33:58.751841228 +0000 UTC m=+856.605749904" Sep 10 14:33:58.784174 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:58.784143 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:33:59.736617 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:59.736580 2576 generic.go:356] "Generic (PLEG): container finished" podID="0bd6ef13-b379-4d77-94cb-3941c90a92ff" containerID="cd190a6327c3620ba8ee67362f5a073260c1df1caba968d30651f8577a12b3f6" exitCode=0 Sep 10 14:33:59.737005 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:33:59.736626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" event={"ID":"0bd6ef13-b379-4d77-94cb-3941c90a92ff","Type":"ContainerDied","Data":"cd190a6327c3620ba8ee67362f5a073260c1df1caba968d30651f8577a12b3f6"} Sep 10 14:34:00.705207 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:00.705173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:00.876596 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:00.876575 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:34:00.907062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:00.907035 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6ef13-b379-4d77-94cb-3941c90a92ff-maas-api-service-ca\") pod \"0bd6ef13-b379-4d77-94cb-3941c90a92ff\" (UID: \"0bd6ef13-b379-4d77-94cb-3941c90a92ff\") " Sep 10 14:34:00.907346 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:00.907323 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd6ef13-b379-4d77-94cb-3941c90a92ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0bd6ef13-b379-4d77-94cb-3941c90a92ff" (UID: "0bd6ef13-b379-4d77-94cb-3941c90a92ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:34:01.008312 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:01.008224 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0bd6ef13-b379-4d77-94cb-3941c90a92ff-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:01.747694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:01.747658 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" event={"ID":"0bd6ef13-b379-4d77-94cb-3941c90a92ff","Type":"ContainerDied","Data":"a8a874c14b94f55da7f1695777e84dbcd9e14190899884bbaf2db07e9ed6940c"} Sep 10 14:34:01.747694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:01.747679 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d" Sep 10 14:34:01.747921 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:01.747709 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a874c14b94f55da7f1695777e84dbcd9e14190899884bbaf2db07e9ed6940c" Sep 10 14:34:04.211651 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:04.211614 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:34:04.212132 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:04.211707 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:34:04.224238 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:04.224217 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:34:04.775990 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:04.775963 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:34:05.012056 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:05.012021 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:05.619346 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:05.619309 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:05.742835 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:05.742751 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:05.843574 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:05.843544 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:05.920488 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:05.920449 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:06.053769 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:06.053727 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:06.735955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:06.735919 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:06.838370 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:06.838340 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:06.941278 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:06.941242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:07.840303 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:07.840264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:08.311992 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.311957 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:08.437461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.437424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:08.537672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.537633 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:08.630667 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.630640 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d"] Sep 10 14:34:08.635485 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.635457 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:08.635972 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.635955 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-d24d4c-hm69d"] Sep 10 14:34:08.706737 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.706708 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:08.761687 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.761653 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bd6ef13-b379-4d77-94cb-3941c90a92ff" path="/var/lib/kubelet/pods/0bd6ef13-b379-4d77-94cb-3941c90a92ff/volumes" Sep 10 14:34:08.973865 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:08.973765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:11.353459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.353417 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" podUID="c6977d85-e827-4442-a85b-056900d690d6" containerName="main" containerID="cri-o://97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4" gracePeriod=45 Sep 10 14:34:11.606205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.606147 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:34:11.707580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707550 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-home\") pod \"c6977d85-e827-4442-a85b-056900d690d6\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " Sep 10 14:34:11.707712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707593 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-tmp-dir\") pod \"c6977d85-e827-4442-a85b-056900d690d6\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " Sep 10 14:34:11.707712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707611 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-model-cache\") pod \"c6977d85-e827-4442-a85b-056900d690d6\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " Sep 10 14:34:11.707712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707661 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzchk\" (UniqueName: \"kubernetes.io/projected/c6977d85-e827-4442-a85b-056900d690d6-kube-api-access-hzchk\") pod \"c6977d85-e827-4442-a85b-056900d690d6\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " Sep 10 14:34:11.707889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707712 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6977d85-e827-4442-a85b-056900d690d6-tls-certs\") pod \"c6977d85-e827-4442-a85b-056900d690d6\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " Sep 10 14:34:11.707889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707773 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-home" (OuterVolumeSpecName: "home") pod "c6977d85-e827-4442-a85b-056900d690d6" (UID: "c6977d85-e827-4442-a85b-056900d690d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:11.707889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707827 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-dshm\") pod \"c6977d85-e827-4442-a85b-056900d690d6\" (UID: \"c6977d85-e827-4442-a85b-056900d690d6\") " Sep 10 14:34:11.707889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707853 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6977d85-e827-4442-a85b-056900d690d6" (UID: "c6977d85-e827-4442-a85b-056900d690d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:11.708033 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.707935 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6977d85-e827-4442-a85b-056900d690d6" (UID: "c6977d85-e827-4442-a85b-056900d690d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:11.708090 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.708072 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-home\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:11.708129 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.708093 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-tmp-dir\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:11.708129 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.708106 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-model-cache\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:11.709949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.709923 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6977d85-e827-4442-a85b-056900d690d6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6977d85-e827-4442-a85b-056900d690d6" (UID: "c6977d85-e827-4442-a85b-056900d690d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:34:11.710059 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.709961 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6977d85-e827-4442-a85b-056900d690d6-kube-api-access-hzchk" (OuterVolumeSpecName: "kube-api-access-hzchk") pod "c6977d85-e827-4442-a85b-056900d690d6" (UID: "c6977d85-e827-4442-a85b-056900d690d6"). InnerVolumeSpecName "kube-api-access-hzchk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:34:11.710059 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.710029 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-dshm" (OuterVolumeSpecName: "dshm") pod "c6977d85-e827-4442-a85b-056900d690d6" (UID: "c6977d85-e827-4442-a85b-056900d690d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:11.798673 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.798638 2576 generic.go:356] "Generic (PLEG): container finished" podID="c6977d85-e827-4442-a85b-056900d690d6" containerID="97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4" exitCode=0 Sep 10 14:34:11.798856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.798678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" Sep 10 14:34:11.798856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.798693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" event={"ID":"c6977d85-e827-4442-a85b-056900d690d6","Type":"ContainerDied","Data":"97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4"} Sep 10 14:34:11.798856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.798732 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl" event={"ID":"c6977d85-e827-4442-a85b-056900d690d6","Type":"ContainerDied","Data":"054e336b79775111284c85e9040e3984652819afe3eec759a1e75845bc9cb4b4"} Sep 10 14:34:11.798856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.798753 2576 scope.go:117] "RemoveContainer" containerID="97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4" Sep 10 14:34:11.807654 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.807638 2576 scope.go:117] "RemoveContainer" containerID="97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4" Sep 10 14:34:11.807921 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:34:11.807903 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4\": container with ID starting with 97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4 not found: ID does not exist" containerID="97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4" Sep 10 14:34:11.807972 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.807930 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4"} err="failed to get container status \"97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4\": rpc error: code = NotFound desc = could not find container \"97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4\": container with ID starting with 97327b655a5409757d3e1c67b18a7836c085df2de1ac533aaaeea71cc6289fc4 not found: ID does not exist" Sep 10 14:34:11.809301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.809279 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6977d85-e827-4442-a85b-056900d690d6-tls-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:11.809301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.809302 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6977d85-e827-4442-a85b-056900d690d6-dshm\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:11.809425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.809310 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzchk\" (UniqueName: \"kubernetes.io/projected/c6977d85-e827-4442-a85b-056900d690d6-kube-api-access-hzchk\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:11.820937 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.820916 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl"] Sep 10 14:34:11.829487 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:11.829462 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-12068cef-kserve-59cd9d6684-zxjhl"] Sep 10 14:34:12.304303 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:12.304267 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:34:12.762135 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:12.762089 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6977d85-e827-4442-a85b-056900d690d6" path="/var/lib/kubelet/pods/c6977d85-e827-4442-a85b-056900d690d6/volumes" Sep 10 14:34:14.853074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:14.853038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:14.959895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:14.959855 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:15.055646 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.055607 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:15.154352 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.154308 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:15.387736 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.387701 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq"] Sep 10 14:34:15.387978 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.387952 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" containerID="cri-o://635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae" gracePeriod=30 Sep 10 14:34:15.418898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.418828 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:15.585735 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.585702 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:15.817622 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:15.817514 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:16.119465 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:16.119375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:16.329256 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:16.329214 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:16.628713 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:16.628676 2576 patch_prober.go:28] interesting pod/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 14:34:16.628905 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:16.628778 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 14:34:19.642396 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:19.642357 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:19.750245 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:19.750208 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:20.640034 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.640010 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:34:20.789035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789007 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-envoy\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789058 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-credential-socket\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789086 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istiod-ca-cert\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789112 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-podinfo\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789132 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-socket\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789173 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nfk5\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-kube-api-access-7nfk5\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789365 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-data\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789439 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-certs\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789471 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-token\") pod \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\" (UID: \"5c7f1a1e-36d9-4249-9a18-ea12b16da199\") " Sep 10 14:34:20.789512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789481 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789516 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789583 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-data" (OuterVolumeSpecName: "istio-data") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789694 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789814 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789833 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789848 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789862 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.790071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.789877 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.791168 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.791147 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:34:20.791572 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.791545 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-token" (OuterVolumeSpecName: "istio-token") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:34:20.791682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.791653 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-kube-api-access-7nfk5" (OuterVolumeSpecName: "kube-api-access-7nfk5") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "kube-api-access-7nfk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:34:20.791682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.791651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5c7f1a1e-36d9-4249-9a18-ea12b16da199" (UID: "5c7f1a1e-36d9-4249-9a18-ea12b16da199"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:34:20.796186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.796162 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:20.846675 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.846641 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerID="635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae" exitCode=0 Sep 10 14:34:20.846866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.846692 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" Sep 10 14:34:20.846866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.846683 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" event={"ID":"5c7f1a1e-36d9-4249-9a18-ea12b16da199","Type":"ContainerDied","Data":"635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae"} Sep 10 14:34:20.846866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.846841 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq" event={"ID":"5c7f1a1e-36d9-4249-9a18-ea12b16da199","Type":"ContainerDied","Data":"e360fa0f9aa532bdbfe7757b65128c1b784619a0d685b7235a47a9d66fdcf467"} Sep 10 14:34:20.847015 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.846870 2576 scope.go:117] "RemoveContainer" containerID="635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae" Sep 10 14:34:20.857189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.857170 2576 scope.go:117] "RemoveContainer" containerID="8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4" Sep 10 14:34:20.865511 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.865490 2576 scope.go:117] "RemoveContainer" containerID="635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae" Sep 10 14:34:20.865811 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:34:20.865770 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae\": container with ID starting with 635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae not found: ID does not exist" containerID="635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae" Sep 10 14:34:20.865879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.865815 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae"} err="failed to get container status \"635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae\": rpc error: code = NotFound desc = could not find container \"635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae\": container with ID starting with 635119c39f71ea1838b56edcebb5617f3dbde400bf24c9dfbbdbf48bc34c35ae not found: ID does not exist" Sep 10 14:34:20.865879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.865835 2576 scope.go:117] "RemoveContainer" containerID="8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4" Sep 10 14:34:20.866096 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:34:20.866076 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4\": container with ID starting with 8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4 not found: ID does not exist" containerID="8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4" Sep 10 14:34:20.866138 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.866103 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4"} err="failed to get container status \"8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4\": rpc error: code = NotFound desc = could not find container \"8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4\": container with ID starting with 8a1a7dfd634e4d8d88cf217242ca9c8b5d65cbc21067cf4cd4bf0e758f2290c4 not found: ID does not exist" Sep 10 14:34:20.871910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.871889 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq"] Sep 10 14:34:20.879162 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.879143 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-d24d4c-openshift-default-7986f99c8d-t2tlq"] Sep 10 14:34:20.880599 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.880574 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:20.890781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.890758 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.890895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.890784 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.890895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.890811 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfk5\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-kube-api-access-7nfk5\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:20.890895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:20.890825 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c7f1a1e-36d9-4249-9a18-ea12b16da199-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:34:21.262133 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:21.262098 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:21.380213 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:21.380174 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:21.636222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:21.636190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:22.762297 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:22.762266 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" path="/var/lib/kubelet/pods/5c7f1a1e-36d9-4249-9a18-ea12b16da199/volumes" Sep 10 14:34:28.613377 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:28.613334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:34:41.469254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:41.469169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:42.421319 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421288 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg"] Sep 10 14:34:42.421632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421620 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" Sep 10 14:34:42.421677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421633 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" Sep 10 14:34:42.421677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421641 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bd6ef13-b379-4d77-94cb-3941c90a92ff" containerName="revoke-keys" Sep 10 14:34:42.421677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421647 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bd6ef13-b379-4d77-94cb-3941c90a92ff" containerName="revoke-keys" Sep 10 14:34:42.421677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421665 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" Sep 10 14:34:42.421677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421672 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" Sep 10 14:34:42.421907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421680 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6977d85-e827-4442-a85b-056900d690d6" containerName="main" Sep 10 14:34:42.421907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421686 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6977d85-e827-4442-a85b-056900d690d6" containerName="main" Sep 10 14:34:42.421907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421737 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bd6ef13-b379-4d77-94cb-3941c90a92ff" containerName="revoke-keys" Sep 10 14:34:42.421907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421744 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" Sep 10 14:34:42.421907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421751 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6977d85-e827-4442-a85b-056900d690d6" containerName="main" Sep 10 14:34:42.421907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.421758 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c7f1a1e-36d9-4249-9a18-ea12b16da199" containerName="istio-proxy" Sep 10 14:34:42.424890 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.424873 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.428760 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.428733 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"default-dockercfg-46rtp\"" Sep 10 14:34:42.428760 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.428737 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"openshift-service-ca.crt\"" Sep 10 14:34:42.428941 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.428760 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"test-model-8ee208-kserve-self-signed-certs\"" Sep 10 14:34:42.428941 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.428768 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"kube-root-ca.crt\"" Sep 10 14:34:42.433631 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.433608 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg"] Sep 10 14:34:42.512680 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.512640 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:42.580997 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.580961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-model-cache\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.581182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.581050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-dshm\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.581182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.581137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-home\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.581182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.581165 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tmp-dir\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.581347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.581213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tls-certs\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.581347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.581247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mqh9\" (UniqueName: \"kubernetes.io/projected/007d8dbe-2e5b-403f-af16-c3ace533d7b3-kube-api-access-4mqh9\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.590418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.590395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:34:42.591150 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.591130 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:34:42.681983 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.681910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-home\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.681983 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.681944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tmp-dir\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.681983 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.681962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tls-certs\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.682204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.681986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mqh9\" (UniqueName: \"kubernetes.io/projected/007d8dbe-2e5b-403f-af16-c3ace533d7b3-kube-api-access-4mqh9\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.682204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.682028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-model-cache\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.682204 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.682059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-dshm\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.682364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.682293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-home\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.682364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.682325 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tmp-dir\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.682453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.682371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-model-cache\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.684196 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.684178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-dshm\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.684766 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.684747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"test-model-8ee208-kserve-self-signed-certs\"" Sep 10 14:34:42.690486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.690465 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"kube-root-ca.crt\"" Sep 10 14:34:42.695466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.695444 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tls-certs\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.701106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.701089 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"openshift-service-ca.crt\"" Sep 10 14:34:42.710932 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.710906 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mqh9\" (UniqueName: \"kubernetes.io/projected/007d8dbe-2e5b-403f-af16-c3ace533d7b3-kube-api-access-4mqh9\") pod \"test-model-8ee208-kserve-74585c8bbf-7hxgg\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:42.737932 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.737914 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8ee208\"/\"default-dockercfg-46rtp\"" Sep 10 14:34:42.746377 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:42.746358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:43.023942 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:43.023893 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg"] Sep 10 14:34:43.026044 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:34:43.026017 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod007d8dbe_2e5b_403f_af16_c3ace533d7b3.slice/crio-816e12ca1a6b8df6c734a4b785e223b19ea0d22adfde4ced766d57d431608ebc WatchSource:0}: Error finding container 816e12ca1a6b8df6c734a4b785e223b19ea0d22adfde4ced766d57d431608ebc: Status 404 returned error can't find the container with id 816e12ca1a6b8df6c734a4b785e223b19ea0d22adfde4ced766d57d431608ebc Sep 10 14:34:43.961856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:43.961815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" event={"ID":"007d8dbe-2e5b-403f-af16-c3ace533d7b3","Type":"ContainerStarted","Data":"9985fa8406fae1e055e18c6ae0f0c807814cbdede5b8126a4a95d6b353924198"} Sep 10 14:34:43.961856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:43.961856 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" event={"ID":"007d8dbe-2e5b-403f-af16-c3ace533d7b3","Type":"ContainerStarted","Data":"816e12ca1a6b8df6c734a4b785e223b19ea0d22adfde4ced766d57d431608ebc"} Sep 10 14:34:43.979247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:43.979199 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" podStartSLOduration=1.979184215 podStartE2EDuration="1.979184215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:34:43.977324131 +0000 UTC m=+901.831232808" watchObservedRunningTime="2026-09-10 14:34:43.979184215 +0000 UTC m=+901.833092891" Sep 10 14:34:44.479358 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:44.479324 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:46.423961 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:46.423919 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:34:47.889268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:47.889228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:48.004087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:48.004052 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:48.089095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:48.089062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:34:52.747149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:52.747116 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:52.747709 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:52.747160 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:52.763399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:52.763371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:34:53.017325 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:34:53.017248 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:35:01.517157 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:01.517123 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:35:03.932970 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:03.932922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:09.610772 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:09.610740 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:09.718339 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:09.718305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:09.810651 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:09.810620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:13.538355 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:13.538322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:16.416574 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:16.416535 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:35:20.323587 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:20.323526 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:21.101694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:21.101583 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:24.660701 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.660662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:24.775493 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.775457 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw"] Sep 10 14:35:24.779068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.779044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:24.781975 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.781952 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:35:24.782414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.782399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:35:24.797582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.797556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw"] Sep 10 14:35:24.863611 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.863583 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6912dc7f-44c6-4b72-98b4-ae11165de5eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw\" (UID: \"6912dc7f-44c6-4b72-98b4-ae11165de5eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:24.964907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.964812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6912dc7f-44c6-4b72-98b4-ae11165de5eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw\" (UID: \"6912dc7f-44c6-4b72-98b4-ae11165de5eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:24.965436 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:24.965417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6912dc7f-44c6-4b72-98b4-ae11165de5eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw\" (UID: \"6912dc7f-44c6-4b72-98b4-ae11165de5eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:25.089941 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:25.089906 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:25.359230 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:25.359075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw"] Sep 10 14:35:25.362564 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:35:25.362537 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6912dc7f_44c6_4b72_98b4_ae11165de5eb.slice/crio-2e0e9eef95f69ae55cace67ca4610447bcf1d15f2bfbe950f95f9700b6b76695 WatchSource:0}: Error finding container 2e0e9eef95f69ae55cace67ca4610447bcf1d15f2bfbe950f95f9700b6b76695: Status 404 returned error can't find the container with id 2e0e9eef95f69ae55cace67ca4610447bcf1d15f2bfbe950f95f9700b6b76695 Sep 10 14:35:26.168860 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:26.168822 2576 generic.go:356] "Generic (PLEG): container finished" podID="6912dc7f-44c6-4b72-98b4-ae11165de5eb" containerID="e3e696b3a80e957ed5d09965cccb102adac15b402d9b06a6c73fb2177160bb9c" exitCode=0 Sep 10 14:35:26.169215 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:26.168914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" event={"ID":"6912dc7f-44c6-4b72-98b4-ae11165de5eb","Type":"ContainerDied","Data":"e3e696b3a80e957ed5d09965cccb102adac15b402d9b06a6c73fb2177160bb9c"} Sep 10 14:35:26.169215 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:26.168963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" event={"ID":"6912dc7f-44c6-4b72-98b4-ae11165de5eb","Type":"ContainerStarted","Data":"2e0e9eef95f69ae55cace67ca4610447bcf1d15f2bfbe950f95f9700b6b76695"} Sep 10 14:35:26.822975 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:26.822940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:26.932151 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:26.932114 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:27.028427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.028394 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:27.122826 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.122699 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:27.313747 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.313717 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:27.386545 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.386519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6912dc7f-44c6-4b72-98b4-ae11165de5eb-maas-api-service-ca\") pod \"6912dc7f-44c6-4b72-98b4-ae11165de5eb\" (UID: \"6912dc7f-44c6-4b72-98b4-ae11165de5eb\") " Sep 10 14:35:27.386889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.386867 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6912dc7f-44c6-4b72-98b4-ae11165de5eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6912dc7f-44c6-4b72-98b4-ae11165de5eb" (UID: "6912dc7f-44c6-4b72-98b4-ae11165de5eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:35:27.487150 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.487109 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6912dc7f-44c6-4b72-98b4-ae11165de5eb-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:35:27.783897 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.783806 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:27.869701 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.869662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:27.969654 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:27.969608 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:28.180755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:28.180724 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" Sep 10 14:35:28.180755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:28.180748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw" event={"ID":"6912dc7f-44c6-4b72-98b4-ae11165de5eb","Type":"ContainerDied","Data":"2e0e9eef95f69ae55cace67ca4610447bcf1d15f2bfbe950f95f9700b6b76695"} Sep 10 14:35:28.181027 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:28.180821 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e0e9eef95f69ae55cace67ca4610447bcf1d15f2bfbe950f95f9700b6b76695" Sep 10 14:35:29.514491 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:29.514446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:33.206285 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:33.206248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:35:35.326524 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.326491 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:35.427263 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.427228 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:35.502226 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.502195 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw"] Sep 10 14:35:35.504681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.504653 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-345a2c-zgsbw"] Sep 10 14:35:35.506249 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.506221 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:35.585617 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.585532 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:35.871679 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:35.871588 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:36.097824 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:36.097774 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:36.174986 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:36.174954 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:36.760656 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:36.760622 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6912dc7f-44c6-4b72-98b4-ae11165de5eb" path="/var/lib/kubelet/pods/6912dc7f-44c6-4b72-98b4-ae11165de5eb/volumes" Sep 10 14:35:36.800923 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:36.800894 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:40.944514 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:40.944473 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:42.579445 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:42.579407 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:42.680019 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:42.679985 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:42.802825 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:42.802776 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:43.069395 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.069363 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k"] Sep 10 14:35:43.069959 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.069938 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6912dc7f-44c6-4b72-98b4-ae11165de5eb" containerName="revoke-keys" Sep 10 14:35:43.070016 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.069963 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6912dc7f-44c6-4b72-98b4-ae11165de5eb" containerName="revoke-keys" Sep 10 14:35:43.070088 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.070075 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6912dc7f-44c6-4b72-98b4-ae11165de5eb" containerName="revoke-keys" Sep 10 14:35:43.073541 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.073520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.076408 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.076383 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e25a5e89-openshift-default-dockercfg-v2rd2\"" Sep 10 14:35:43.088164 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.087514 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k"] Sep 10 14:35:43.111031 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.110998 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:43.127755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.127726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-certs\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.127888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.127761 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbrm\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-kube-api-access-dbbrm\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.127888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.127876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab34dde3-223d-41ac-9540-2e0c9645e76f-istiod-ca-cert\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.127966 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.127916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-socket\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.127966 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.127940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-credential-socket\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.128058 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.128042 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-data\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.128094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.128069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-token\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.128128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.128091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-podinfo\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.128161 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.128129 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-envoy\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229113 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-credential-socket\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-data\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-token\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-podinfo\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-envoy\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229504 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-certs\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbrm\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-kube-api-access-dbbrm\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab34dde3-223d-41ac-9540-2e0c9645e76f-istiod-ca-cert\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229583 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-credential-socket\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.229712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.229661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-socket\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.230062 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.230041 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-socket\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.230243 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.230215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-certs\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.230467 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.230433 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-data\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.230548 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.230514 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab34dde3-223d-41ac-9540-2e0c9645e76f-istiod-ca-cert\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.232007 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.231983 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-envoy\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.232561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.232542 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-podinfo\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.252753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.252725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbrm\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-kube-api-access-dbbrm\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.255502 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.255285 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:43.255502 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.255451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-token\") pod \"e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.390641 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.390604 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:43.665205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.665175 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k"] Sep 10 14:35:43.666932 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:35:43.666905 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab34dde3_223d_41ac_9540_2e0c9645e76f.slice/crio-69c31c88bb497e6df1c3296ee391260ab6b532cb7ece3567b44062104f517475 WatchSource:0}: Error finding container 69c31c88bb497e6df1c3296ee391260ab6b532cb7ece3567b44062104f517475: Status 404 returned error can't find the container with id 69c31c88bb497e6df1c3296ee391260ab6b532cb7ece3567b44062104f517475 Sep 10 14:35:43.669521 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.669490 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:35:43.669576 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.669556 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:35:43.669614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:43.669582 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:35:44.267344 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:44.267302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" event={"ID":"ab34dde3-223d-41ac-9540-2e0c9645e76f","Type":"ContainerStarted","Data":"fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd"} Sep 10 14:35:44.267529 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:44.267361 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" event={"ID":"ab34dde3-223d-41ac-9540-2e0c9645e76f","Type":"ContainerStarted","Data":"69c31c88bb497e6df1c3296ee391260ab6b532cb7ece3567b44062104f517475"} Sep 10 14:35:44.287920 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:44.287874 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podStartSLOduration=1.287859426 podStartE2EDuration="1.287859426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:44.286535123 +0000 UTC m=+962.140443800" watchObservedRunningTime="2026-09-10 14:35:44.287859426 +0000 UTC m=+962.141768145" Sep 10 14:35:44.390771 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:44.390736 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:44.395910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:44.395884 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:45.272739 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:45.272711 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:45.273630 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:45.273607 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:35:46.215251 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:46.215207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:46.317322 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:46.317279 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:47.397725 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:47.397692 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:47.493723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:47.493670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:47.523111 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:47.523083 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:35:47.613112 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:47.613081 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:47.903383 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:47.903351 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:48.003445 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.003410 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:48.084372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.084342 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:48.182336 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.182251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:48.593188 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.593096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:48.869848 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.869751 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5"] Sep 10 14:35:48.873565 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.873544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.876528 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.876500 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-651c34-openshift-default-dockercfg-ttfz6\"" Sep 10 14:35:48.887659 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.887637 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5"] Sep 10 14:35:48.906166 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.906134 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:48.983769 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.983737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-envoy\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.983959 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.983823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-istio-token\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.983959 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.983923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-certs\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.984082 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.983987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092423a7-9333-4683-ba9d-d261263f9359-istiod-ca-cert\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.984082 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.984023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-data\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.984082 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.984073 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-credential-socket\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.984224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.984101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092423a7-9333-4683-ba9d-d261263f9359-istio-podinfo\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.984257 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.984220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-socket\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:48.984294 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:48.984262 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8b2\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-kube-api-access-tq8b2\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.038641 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.038606 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:49.085542 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-credential-socket\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.085696 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092423a7-9333-4683-ba9d-d261263f9359-istio-podinfo\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.085696 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085652 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-socket\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.085853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085694 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8b2\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-kube-api-access-tq8b2\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.085853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-envoy\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.085853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-istio-token\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.085853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-certs\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086058 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085902 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092423a7-9333-4683-ba9d-d261263f9359-istiod-ca-cert\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086058 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085958 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-credential-socket\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086058 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.085997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-data\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086213 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.086146 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-socket\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086340 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.086317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-data\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086408 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.086368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-certs\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.086506 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.086489 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092423a7-9333-4683-ba9d-d261263f9359-istiod-ca-cert\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.088549 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.088526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-envoy\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.088750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.088735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092423a7-9333-4683-ba9d-d261263f9359-istio-podinfo\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.093580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.093558 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8b2\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-kube-api-access-tq8b2\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.094150 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.094084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-istio-token\") pod \"e2e-noten-651c34-openshift-default-7777797c4d-j98l5\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.185913 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.185883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:49.464355 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.464322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5"] Sep 10 14:35:49.467600 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:35:49.467567 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092423a7_9333_4683_ba9d_d261263f9359.slice/crio-11642b8fbf10c828e4d6e0035924b3e585bc9d012e695399792ae9096d0556e0 WatchSource:0}: Error finding container 11642b8fbf10c828e4d6e0035924b3e585bc9d012e695399792ae9096d0556e0: Status 404 returned error can't find the container with id 11642b8fbf10c828e4d6e0035924b3e585bc9d012e695399792ae9096d0556e0 Sep 10 14:35:49.470588 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.470551 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:35:49.470692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.470635 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:35:49.470692 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:49.470681 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:35:50.299165 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:50.299130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" event={"ID":"092423a7-9333-4683-ba9d-d261263f9359","Type":"ContainerStarted","Data":"b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43"} Sep 10 14:35:50.299633 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:50.299183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" event={"ID":"092423a7-9333-4683-ba9d-d261263f9359","Type":"ContainerStarted","Data":"11642b8fbf10c828e4d6e0035924b3e585bc9d012e695399792ae9096d0556e0"} Sep 10 14:35:50.321179 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:50.321128 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" podStartSLOduration=2.321113038 podStartE2EDuration="2.321113038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:35:50.319383491 +0000 UTC m=+968.173292181" watchObservedRunningTime="2026-09-10 14:35:50.321113038 +0000 UTC m=+968.175021714" Sep 10 14:35:50.591393 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:50.591303 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:50.683511 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:50.683469 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:50.801604 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:50.801571 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:35:51.186213 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:51.186182 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:51.191144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:51.191120 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:51.303834 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:51.303811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:35:51.304606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:35:51.304587 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:36:03.206043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:03.205997 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:36:05.378397 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:05.378363 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerID="fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd" exitCode=0 Sep 10 14:36:05.378940 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:05.378440 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" event={"ID":"ab34dde3-223d-41ac-9540-2e0c9645e76f","Type":"ContainerDied","Data":"fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd"} Sep 10 14:36:05.379034 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:05.379016 2576 scope.go:117] "RemoveContainer" containerID="fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd" Sep 10 14:36:05.380093 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:05.380064 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:36:05.380186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:05.380133 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:36:05.380186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:05.380168 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:36:06.387073 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:06.387034 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" event={"ID":"ab34dde3-223d-41ac-9540-2e0c9645e76f","Type":"ContainerStarted","Data":"9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf"} Sep 10 14:36:06.390817 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:06.390779 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:36:06.391047 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:06.391023 2576 patch_prober.go:28] interesting pod/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 14:36:06.391097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:06.391072 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 14:36:07.391451 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:07.391422 2576 patch_prober.go:28] interesting pod/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 14:36:07.391859 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:07.391474 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 14:36:08.391955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:08.391918 2576 patch_prober.go:28] interesting pod/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 14:36:08.392370 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:08.392024 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 14:36:09.278044 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:09.278001 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:09.391516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:09.391477 2576 patch_prober.go:28] interesting pod/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 14:36:09.391691 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:09.391536 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 14:36:10.391176 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:10.391135 2576 patch_prober.go:28] interesting pod/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Sep 10 14:36:10.391598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:10.391201 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Sep 10 14:36:11.171556 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.171513 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:11.584445 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.584362 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:36:11.585178 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.585136 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:36:11.588757 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.588735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:36:11.835324 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.835238 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:11.906278 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.906237 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp"] Sep 10 14:36:11.907614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.907573 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:11.978209 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:11.978175 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:12.274314 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:12.274276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:18.134219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:18.134183 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:18.189241 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:18.189199 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:21.803851 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:21.803821 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:36:22.362385 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:22.362353 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg"] Sep 10 14:36:22.380249 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:22.380217 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:22.620161 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:22.620067 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:23.411260 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:23.411218 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:23.509480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:23.509448 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5"] Sep 10 14:36:23.509712 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:23.509685 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" podUID="092423a7-9333-4683-ba9d-d261263f9359" containerName="istio-proxy" containerID="cri-o://b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43" gracePeriod=30 Sep 10 14:36:23.536893 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:23.536863 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:23.763503 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:23.763408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:23.876946 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:23.876909 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:26.927217 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:26.927165 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" podUID="72405153-6a4b-46a9-91e5-27f9a3255a83" containerName="main" containerID="cri-o://daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3" gracePeriod=45 Sep 10 14:36:27.185733 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.185678 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:36:27.248048 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248016 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-dshm\") pod \"72405153-6a4b-46a9-91e5-27f9a3255a83\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " Sep 10 14:36:27.248189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248065 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-home\") pod \"72405153-6a4b-46a9-91e5-27f9a3255a83\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " Sep 10 14:36:27.248189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248091 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t446f\" (UniqueName: \"kubernetes.io/projected/72405153-6a4b-46a9-91e5-27f9a3255a83-kube-api-access-t446f\") pod \"72405153-6a4b-46a9-91e5-27f9a3255a83\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " Sep 10 14:36:27.248189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248116 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-model-cache\") pod \"72405153-6a4b-46a9-91e5-27f9a3255a83\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " Sep 10 14:36:27.248189 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248139 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72405153-6a4b-46a9-91e5-27f9a3255a83-tls-certs\") pod \"72405153-6a4b-46a9-91e5-27f9a3255a83\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " Sep 10 14:36:27.248399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248238 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-tmp-dir\") pod \"72405153-6a4b-46a9-91e5-27f9a3255a83\" (UID: \"72405153-6a4b-46a9-91e5-27f9a3255a83\") " Sep 10 14:36:27.248458 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248419 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-home" (OuterVolumeSpecName: "home") pod "72405153-6a4b-46a9-91e5-27f9a3255a83" (UID: "72405153-6a4b-46a9-91e5-27f9a3255a83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:27.248458 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248431 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-model-cache" (OuterVolumeSpecName: "model-cache") pod "72405153-6a4b-46a9-91e5-27f9a3255a83" (UID: "72405153-6a4b-46a9-91e5-27f9a3255a83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:27.248619 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248593 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "72405153-6a4b-46a9-91e5-27f9a3255a83" (UID: "72405153-6a4b-46a9-91e5-27f9a3255a83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:27.248694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248604 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-model-cache\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:27.248694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.248661 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-home\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:27.250419 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.250393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72405153-6a4b-46a9-91e5-27f9a3255a83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "72405153-6a4b-46a9-91e5-27f9a3255a83" (UID: "72405153-6a4b-46a9-91e5-27f9a3255a83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:36:27.250714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.250693 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-dshm" (OuterVolumeSpecName: "dshm") pod "72405153-6a4b-46a9-91e5-27f9a3255a83" (UID: "72405153-6a4b-46a9-91e5-27f9a3255a83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:27.250714 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.250706 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72405153-6a4b-46a9-91e5-27f9a3255a83-kube-api-access-t446f" (OuterVolumeSpecName: "kube-api-access-t446f") pod "72405153-6a4b-46a9-91e5-27f9a3255a83" (UID: "72405153-6a4b-46a9-91e5-27f9a3255a83"). InnerVolumeSpecName "kube-api-access-t446f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:36:27.349783 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.349745 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-dshm\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:27.349964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.349787 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t446f\" (UniqueName: \"kubernetes.io/projected/72405153-6a4b-46a9-91e5-27f9a3255a83-kube-api-access-t446f\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:27.349964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.349930 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72405153-6a4b-46a9-91e5-27f9a3255a83-tls-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:27.349964 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.349944 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72405153-6a4b-46a9-91e5-27f9a3255a83-tmp-dir\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:27.504284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.504187 2576 generic.go:356] "Generic (PLEG): container finished" podID="72405153-6a4b-46a9-91e5-27f9a3255a83" containerID="daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3" exitCode=0 Sep 10 14:36:27.504284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.504229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" event={"ID":"72405153-6a4b-46a9-91e5-27f9a3255a83","Type":"ContainerDied","Data":"daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3"} Sep 10 14:36:27.504284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.504247 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" Sep 10 14:36:27.504284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.504258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp" event={"ID":"72405153-6a4b-46a9-91e5-27f9a3255a83","Type":"ContainerDied","Data":"f816ae1d5d15e3e045e330417ba671cc26a76b5c2e7b24194a698dc7d1b6071d"} Sep 10 14:36:27.504284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.504278 2576 scope.go:117] "RemoveContainer" containerID="daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3" Sep 10 14:36:27.513060 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.513038 2576 scope.go:117] "RemoveContainer" containerID="daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3" Sep 10 14:36:27.513327 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:36:27.513305 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3\": container with ID starting with daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3 not found: ID does not exist" containerID="daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3" Sep 10 14:36:27.513392 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.513334 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3"} err="failed to get container status \"daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3\": rpc error: code = NotFound desc = could not find container \"daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3\": container with ID starting with daec093c6e03d3bf53a841291cbdd433eab33d512ef7ee880a3dbb60cd000bf3 not found: ID does not exist" Sep 10 14:36:27.535670 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.535643 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp"] Sep 10 14:36:27.543524 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:27.543497 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-600197/test-model-600197-kserve-586597d96c-9wxpp"] Sep 10 14:36:28.723952 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.723906 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:28.761382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.761349 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72405153-6a4b-46a9-91e5-27f9a3255a83" path="/var/lib/kubelet/pods/72405153-6a4b-46a9-91e5-27f9a3255a83/volumes" Sep 10 14:36:28.761882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.761861 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:36:28.862960 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.862880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-certs\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.862960 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.862927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-envoy\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.862971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092423a7-9333-4683-ba9d-d261263f9359-istiod-ca-cert\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.862994 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-data\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-socket\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863101 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092423a7-9333-4683-ba9d-d261263f9359-istio-podinfo\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863165 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq8b2\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-kube-api-access-tq8b2\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863177 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:28.863427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863195 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-credential-socket\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863427 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863234 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-istio-token\") pod \"092423a7-9333-4683-ba9d-d261263f9359\" (UID: \"092423a7-9333-4683-ba9d-d261263f9359\") " Sep 10 14:36:28.863575 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863469 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:28.863575 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863490 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/092423a7-9333-4683-ba9d-d261263f9359-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:36:28.863688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:28.863688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863656 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.863826 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863687 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/092423a7-9333-4683-ba9d-d261263f9359-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.863826 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.863715 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.864042 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.864006 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-data" (OuterVolumeSpecName: "istio-data") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:28.865633 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.865612 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-istio-token" (OuterVolumeSpecName: "istio-token") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:36:28.865730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.865631 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:28.865730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.865631 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/092423a7-9333-4683-ba9d-d261263f9359-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:36:28.865928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.865912 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-kube-api-access-tq8b2" (OuterVolumeSpecName: "kube-api-access-tq8b2") pod "092423a7-9333-4683-ba9d-d261263f9359" (UID: "092423a7-9333-4683-ba9d-d261263f9359"). InnerVolumeSpecName "kube-api-access-tq8b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:36:28.954652 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.954619 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:28.964903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.964877 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.965022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.964905 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/092423a7-9333-4683-ba9d-d261263f9359-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.965022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.964920 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq8b2\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-kube-api-access-tq8b2\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.965022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.964936 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/092423a7-9333-4683-ba9d-d261263f9359-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.965022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.964950 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:28.965022 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:28.964962 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/092423a7-9333-4683-ba9d-d261263f9359-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:29.255866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.255825 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:29.352084 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.352046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:29.435452 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.435404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:29.499823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.499775 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:29.518920 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.518822 2576 generic.go:356] "Generic (PLEG): container finished" podID="092423a7-9333-4683-ba9d-d261263f9359" containerID="b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43" exitCode=0 Sep 10 14:36:29.519095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.518931 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" Sep 10 14:36:29.519095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.518940 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" event={"ID":"092423a7-9333-4683-ba9d-d261263f9359","Type":"ContainerDied","Data":"b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43"} Sep 10 14:36:29.519095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.519011 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5" event={"ID":"092423a7-9333-4683-ba9d-d261263f9359","Type":"ContainerDied","Data":"11642b8fbf10c828e4d6e0035924b3e585bc9d012e695399792ae9096d0556e0"} Sep 10 14:36:29.519095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.519045 2576 scope.go:117] "RemoveContainer" containerID="b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43" Sep 10 14:36:29.530518 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.530497 2576 scope.go:117] "RemoveContainer" containerID="b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43" Sep 10 14:36:29.530854 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:36:29.530834 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43\": container with ID starting with b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43 not found: ID does not exist" containerID="b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43" Sep 10 14:36:29.530934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.530863 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43"} err="failed to get container status \"b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43\": rpc error: code = NotFound desc = could not find container \"b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43\": container with ID starting with b356a8ef321a37a40ea6fd694428b07e106f55ab5a2137c9b178731565f4fc43 not found: ID does not exist" Sep 10 14:36:29.543701 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.543673 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5"] Sep 10 14:36:29.547700 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.547673 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-651c34-openshift-default-7777797c4d-j98l5"] Sep 10 14:36:29.788494 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:29.788403 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:30.761810 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:30.761767 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="092423a7-9333-4683-ba9d-d261263f9359" path="/var/lib/kubelet/pods/092423a7-9333-4683-ba9d-d261263f9359/volumes" Sep 10 14:36:31.918232 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:31.918196 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:36:32.608870 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:32.608834 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:33.138850 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:33.138809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:33.200949 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:33.200908 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:33.258217 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:33.258169 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:33.323447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:33.323409 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:33.678413 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:33.678382 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:35.715308 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:35.715272 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:35.916380 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:35.916341 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:35.961083 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:35.961053 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k"] Sep 10 14:36:35.961321 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:35.961296 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" containerID="cri-o://9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf" gracePeriod=30 Sep 10 14:36:36.016659 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:36.016568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:36.306345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:36.306258 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:36.402276 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:36.402242 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:36.503163 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:36.503124 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:37.383113 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.383070 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" podUID="007d8dbe-2e5b-403f-af16-c3ace533d7b3" containerName="main" containerID="cri-o://9985fa8406fae1e055e18c6ae0f0c807814cbdede5b8126a4a95d6b353924198" gracePeriod=45 Sep 10 14:36:37.565518 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.565482 2576 generic.go:356] "Generic (PLEG): container finished" podID="007d8dbe-2e5b-403f-af16-c3ace533d7b3" containerID="9985fa8406fae1e055e18c6ae0f0c807814cbdede5b8126a4a95d6b353924198" exitCode=0 Sep 10 14:36:37.565657 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.565552 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" event={"ID":"007d8dbe-2e5b-403f-af16-c3ace533d7b3","Type":"ContainerDied","Data":"9985fa8406fae1e055e18c6ae0f0c807814cbdede5b8126a4a95d6b353924198"} Sep 10 14:36:37.637382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.637324 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:36:37.751972 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.751943 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-model-cache\") pod \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " Sep 10 14:36:37.752149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.751990 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tmp-dir\") pod \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " Sep 10 14:36:37.752149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752017 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tls-certs\") pod \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " Sep 10 14:36:37.752149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752077 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-dshm\") pod \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " Sep 10 14:36:37.752149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752140 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-home\") pod \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " Sep 10 14:36:37.752388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "007d8dbe-2e5b-403f-af16-c3ace533d7b3" (UID: "007d8dbe-2e5b-403f-af16-c3ace533d7b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:37.752388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752176 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mqh9\" (UniqueName: \"kubernetes.io/projected/007d8dbe-2e5b-403f-af16-c3ace533d7b3-kube-api-access-4mqh9\") pod \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\" (UID: \"007d8dbe-2e5b-403f-af16-c3ace533d7b3\") " Sep 10 14:36:37.752388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752278 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "007d8dbe-2e5b-403f-af16-c3ace533d7b3" (UID: "007d8dbe-2e5b-403f-af16-c3ace533d7b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:37.752537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752392 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-home" (OuterVolumeSpecName: "home") pod "007d8dbe-2e5b-403f-af16-c3ace533d7b3" (UID: "007d8dbe-2e5b-403f-af16-c3ace533d7b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:37.752537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752499 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-home\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:37.752537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752522 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-model-cache\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:37.752653 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.752537 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tmp-dir\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:37.754398 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.754368 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "007d8dbe-2e5b-403f-af16-c3ace533d7b3" (UID: "007d8dbe-2e5b-403f-af16-c3ace533d7b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:36:37.754398 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.754379 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-dshm" (OuterVolumeSpecName: "dshm") pod "007d8dbe-2e5b-403f-af16-c3ace533d7b3" (UID: "007d8dbe-2e5b-403f-af16-c3ace533d7b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:37.754551 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.754430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007d8dbe-2e5b-403f-af16-c3ace533d7b3-kube-api-access-4mqh9" (OuterVolumeSpecName: "kube-api-access-4mqh9") pod "007d8dbe-2e5b-403f-af16-c3ace533d7b3" (UID: "007d8dbe-2e5b-403f-af16-c3ace533d7b3"). InnerVolumeSpecName "kube-api-access-4mqh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:36:37.852994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.852965 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/007d8dbe-2e5b-403f-af16-c3ace533d7b3-dshm\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:37.852994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.852990 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mqh9\" (UniqueName: \"kubernetes.io/projected/007d8dbe-2e5b-403f-af16-c3ace533d7b3-kube-api-access-4mqh9\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:37.853162 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:37.853000 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/007d8dbe-2e5b-403f-af16-c3ace533d7b3-tls-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:38.573235 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:38.573203 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" Sep 10 14:36:38.573235 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:38.573217 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg" event={"ID":"007d8dbe-2e5b-403f-af16-c3ace533d7b3","Type":"ContainerDied","Data":"816e12ca1a6b8df6c734a4b785e223b19ea0d22adfde4ced766d57d431608ebc"} Sep 10 14:36:38.573729 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:38.573270 2576 scope.go:117] "RemoveContainer" containerID="9985fa8406fae1e055e18c6ae0f0c807814cbdede5b8126a4a95d6b353924198" Sep 10 14:36:38.615162 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:38.615121 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg"] Sep 10 14:36:38.620116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:38.620090 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8ee208/test-model-8ee208-kserve-74585c8bbf-7hxgg"] Sep 10 14:36:38.762603 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:38.762568 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="007d8dbe-2e5b-403f-af16-c3ace533d7b3" path="/var/lib/kubelet/pods/007d8dbe-2e5b-403f-af16-c3ace533d7b3/volumes" Sep 10 14:36:39.494291 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:39.494256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:39.595582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:39.595543 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:39.744203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:39.744164 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:40.309546 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:40.309512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:41.214937 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.214914 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:36:41.382771 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382736 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-envoy\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.382771 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382776 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab34dde3-223d-41ac-9540-2e0c9645e76f-istiod-ca-cert\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382821 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-certs\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382872 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbbrm\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-kube-api-access-dbbrm\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382902 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-token\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-credential-socket\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.382968 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-data\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383003 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-socket\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383073 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-podinfo\") pod \"ab34dde3-223d-41ac-9540-2e0c9645e76f\" (UID: \"ab34dde3-223d-41ac-9540-2e0c9645e76f\") " Sep 10 14:36:41.383343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383208 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:41.383343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383285 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab34dde3-223d-41ac-9540-2e0c9645e76f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:36:41.383498 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383383 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:41.383498 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383487 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:41.383610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383498 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:41.383610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383563 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.383610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383575 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab34dde3-223d-41ac-9540-2e0c9645e76f-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.383610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.383584 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.385100 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.385072 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-kube-api-access-dbbrm" (OuterVolumeSpecName: "kube-api-access-dbbrm") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "kube-api-access-dbbrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:36:41.385200 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.385097 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:36:41.385200 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.385101 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:36:41.385281 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.385216 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab34dde3-223d-41ac-9540-2e0c9645e76f" (UID: "ab34dde3-223d-41ac-9540-2e0c9645e76f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:36:41.484656 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.484625 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.484656 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.484650 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbrm\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-kube-api-access-dbbrm\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.484656 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.484659 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.484915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.484669 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.484915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.484677 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab34dde3-223d-41ac-9540-2e0c9645e76f-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.484915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.484686 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab34dde3-223d-41ac-9540-2e0c9645e76f-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:36:41.591192 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.591154 2576 generic.go:356] "Generic (PLEG): container finished" podID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerID="9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf" exitCode=0 Sep 10 14:36:41.591372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.591217 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" Sep 10 14:36:41.591372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.591251 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" event={"ID":"ab34dde3-223d-41ac-9540-2e0c9645e76f","Type":"ContainerDied","Data":"9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf"} Sep 10 14:36:41.591372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.591313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k" event={"ID":"ab34dde3-223d-41ac-9540-2e0c9645e76f","Type":"ContainerDied","Data":"69c31c88bb497e6df1c3296ee391260ab6b532cb7ece3567b44062104f517475"} Sep 10 14:36:41.591372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.591337 2576 scope.go:117] "RemoveContainer" containerID="9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf" Sep 10 14:36:41.600350 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.600333 2576 scope.go:117] "RemoveContainer" containerID="fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd" Sep 10 14:36:41.608023 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.608007 2576 scope.go:117] "RemoveContainer" containerID="9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf" Sep 10 14:36:41.608287 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:36:41.608269 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf\": container with ID starting with 9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf not found: ID does not exist" containerID="9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf" Sep 10 14:36:41.608380 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.608297 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf"} err="failed to get container status \"9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf\": rpc error: code = NotFound desc = could not find container \"9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf\": container with ID starting with 9a6bb52231dc87a3270d62363a32b0207c7f1909eb67a12bffc5c920dcb39dbf not found: ID does not exist" Sep 10 14:36:41.608380 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.608320 2576 scope.go:117] "RemoveContainer" containerID="fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd" Sep 10 14:36:41.608583 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:36:41.608567 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd\": container with ID starting with fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd not found: ID does not exist" containerID="fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd" Sep 10 14:36:41.608627 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.608587 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd"} err="failed to get container status \"fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd\": rpc error: code = NotFound desc = could not find container \"fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd\": container with ID starting with fb035c74de0a9a915bb709c28ba6c7e6cf0250ae87cde0ffb6653c63d3bf7ccd not found: ID does not exist" Sep 10 14:36:41.612975 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.612953 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k"] Sep 10 14:36:41.617536 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:41.617515 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e25a5e89-openshift-default-678b5bf964-j2n2k"] Sep 10 14:36:42.106975 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:42.106923 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:42.281524 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:42.281479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:42.367192 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:42.367105 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:42.467226 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:42.467193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:42.761944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:42.761911 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" path="/var/lib/kubelet/pods/ab34dde3-223d-41ac-9540-2e0c9645e76f/volumes" Sep 10 14:36:43.308703 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:43.308666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:36:44.702117 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:44.702062 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:48.265933 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:48.265901 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:48.369549 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:48.369512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:49.189816 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:49.189763 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:49.478224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:49.478131 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:49.777263 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:49.777176 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:50.635123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:50.635080 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:50.714888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:50.714851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:50.799932 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:50.799896 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:50.900455 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:50.900370 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:51.203293 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:51.203190 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:52.720375 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:52.720345 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:52.814765 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:52.814730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:53.009274 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:53.009197 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:36:53.848820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:53.848770 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:53.949536 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:53.949504 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:54.282661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:54.282624 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:54.383452 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:54.383421 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:54.446900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:54.446859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:57.290698 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:57.290663 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:57.358674 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:57.358638 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:57.501076 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:57.501046 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w"] Sep 10 14:36:57.501276 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:57.501253 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" containerID="cri-o://1f265db57379eb8cf9cafe93348b302abc8476132ac8abaf1fc30b9df56fff76" gracePeriod=30 Sep 10 14:36:57.527092 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:57.527060 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:58.064024 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:58.063975 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:58.360727 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:58.360643 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:36:58.542623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:36:58.542587 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:01.448510 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:01.448476 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:01.555661 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:01.555622 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:01.809023 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:01.808943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:37:02.630602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.630566 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:02.700610 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.700580 2576 generic.go:356] "Generic (PLEG): container finished" podID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerID="1f265db57379eb8cf9cafe93348b302abc8476132ac8abaf1fc30b9df56fff76" exitCode=0 Sep 10 14:37:02.700774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.700653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" event={"ID":"1a17e847-ed26-4b0d-8c6e-35197ce5b683","Type":"ContainerDied","Data":"1f265db57379eb8cf9cafe93348b302abc8476132ac8abaf1fc30b9df56fff76"} Sep 10 14:37:02.700774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.700717 2576 scope.go:117] "RemoveContainer" containerID="864353a248ff63c688d36ac3719527ec8284265d447095c947e6ba41b8b86532" Sep 10 14:37:02.735164 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.735132 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:02.753138 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.753107 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:37:02.832144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.832046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:02.874138 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874103 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-credential-socket\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874156 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istiod-ca-cert\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874208 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-socket\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7htk\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-kube-api-access-b7htk\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874264 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-certs\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874280 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-envoy\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874324 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-podinfo\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874345 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-data\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874380 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-token\") pod \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\" (UID: \"1a17e847-ed26-4b0d-8c6e-35197ce5b683\") " Sep 10 14:37:02.874537 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874402 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874567 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874582 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874609 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874678 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874696 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874710 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.874740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874720 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.875128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.874825 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:37:02.876620 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.876595 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:37:02.876719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.876619 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:37:02.876719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.876635 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-kube-api-access-b7htk" (OuterVolumeSpecName: "kube-api-access-b7htk") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "kube-api-access-b7htk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:37:02.876719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.876703 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a17e847-ed26-4b0d-8c6e-35197ce5b683" (UID: "1a17e847-ed26-4b0d-8c6e-35197ce5b683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:37:02.976087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.976050 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.976087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.976083 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.976087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.976092 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.976335 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.976102 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:02.976335 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:02.976110 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7htk\" (UniqueName: \"kubernetes.io/projected/1a17e847-ed26-4b0d-8c6e-35197ce5b683-kube-api-access-b7htk\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:37:03.241564 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.241523 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:03.322626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.322586 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:03.344274 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344239 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf"] Sep 10 14:37:03.344667 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344669 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344682 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344687 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344694 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344699 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344709 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="007d8dbe-2e5b-403f-af16-c3ace533d7b3" containerName="main" Sep 10 14:37:03.344717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344718 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="007d8dbe-2e5b-403f-af16-c3ace533d7b3" containerName="main" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344728 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="092423a7-9333-4683-ba9d-d261263f9359" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344734 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="092423a7-9333-4683-ba9d-d261263f9359" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344743 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72405153-6a4b-46a9-91e5-27f9a3255a83" containerName="main" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344748 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="72405153-6a4b-46a9-91e5-27f9a3255a83" containerName="main" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344754 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344759 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344829 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344837 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="007d8dbe-2e5b-403f-af16-c3ace533d7b3" containerName="main" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344844 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344851 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344856 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab34dde3-223d-41ac-9540-2e0c9645e76f" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344862 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="092423a7-9333-4683-ba9d-d261263f9359" containerName="istio-proxy" Sep 10 14:37:03.344955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.344869 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="72405153-6a4b-46a9-91e5-27f9a3255a83" containerName="main" Sep 10 14:37:03.348123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.348106 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.350766 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.350734 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-7bf16f-dockercfg-8p5qp\"" Sep 10 14:37:03.350918 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.350826 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7bf16f\"" Sep 10 14:37:03.358509 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.358484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf"] Sep 10 14:37:03.482068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.482024 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.482243 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.482163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjkq8\" (UniqueName: \"kubernetes.io/projected/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-kube-api-access-fjkq8\") pod \"payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.583145 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.583056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjkq8\" (UniqueName: \"kubernetes.io/projected/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-kube-api-access-fjkq8\") pod \"payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.583145 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.583109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.583867 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.583846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.598551 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.598526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjkq8\" (UniqueName: \"kubernetes.io/projected/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-kube-api-access-fjkq8\") pod \"payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.660489 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.660455 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:03.708437 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.708407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" event={"ID":"1a17e847-ed26-4b0d-8c6e-35197ce5b683","Type":"ContainerDied","Data":"c9ce1336f37c0770b5d90cf76d978c4231b234d459bad95b03b0eababe36eca8"} Sep 10 14:37:03.708521 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.708469 2576 scope.go:117] "RemoveContainer" containerID="1f265db57379eb8cf9cafe93348b302abc8476132ac8abaf1fc30b9df56fff76" Sep 10 14:37:03.708574 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.708532 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w" Sep 10 14:37:03.753523 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.753495 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w"] Sep 10 14:37:03.756066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.756043 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8ee208-openshift-default-76f8b95d56-n5g5w"] Sep 10 14:37:03.933365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.933265 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf"] Sep 10 14:37:03.935499 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:37:03.935469 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1991246_c7c1_41f5_b6af_d0b5d55db3fc.slice/crio-86ebe4a48e2641899aed23e1b7b4b7467d0b5f90f2040620c32605217565f0c0 WatchSource:0}: Error finding container 86ebe4a48e2641899aed23e1b7b4b7467d0b5f90f2040620c32605217565f0c0: Status 404 returned error can't find the container with id 86ebe4a48e2641899aed23e1b7b4b7467d0b5f90f2040620c32605217565f0c0 Sep 10 14:37:03.937574 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:03.937556 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 14:37:04.715160 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:04.715125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" event={"ID":"a1991246-c7c1-41f5-b6af-d0b5d55db3fc","Type":"ContainerStarted","Data":"86ebe4a48e2641899aed23e1b7b4b7467d0b5f90f2040620c32605217565f0c0"} Sep 10 14:37:04.767416 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:04.767376 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a17e847-ed26-4b0d-8c6e-35197ce5b683" path="/var/lib/kubelet/pods/1a17e847-ed26-4b0d-8c6e-35197ce5b683/volumes" Sep 10 14:37:05.722741 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:05.722709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" event={"ID":"a1991246-c7c1-41f5-b6af-d0b5d55db3fc","Type":"ContainerStarted","Data":"ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f"} Sep 10 14:37:05.723115 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:05.722840 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:05.741301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:05.741249 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" podStartSLOduration=1.178688587 podStartE2EDuration="2.741233534s" totalImagesPullingTime="1.562544947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:05.738833297 +0000 UTC m=+1043.592741968" watchObservedRunningTime="2026-09-10 14:37:05.741233534 +0000 UTC m=+1043.595142241" Sep 10 14:37:05.939242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:05.939207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:10.305614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:10.305579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:37:14.791584 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:14.791547 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:16.731297 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:16.731265 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:37:17.662642 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:17.662610 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:18.064695 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.064604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:18.273313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.273281 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc"] Sep 10 14:37:18.276837 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.276814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.279580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.279550 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-fc06d4-openshift-default-dockercfg-xmsf8\"" Sep 10 14:37:18.288405 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.288380 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc"] Sep 10 14:37:18.302757 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.302730 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:18.410784 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-socket\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.410784 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjpcp\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-kube-api-access-mjpcp\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-token\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-data\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-envoy\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istiod-ca-cert\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.410960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-podinfo\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.411027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-certs\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.411190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.411069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-credential-socket\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.492194 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.492155 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:18.512219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-certs\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-credential-socket\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-socket\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjpcp\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-kube-api-access-mjpcp\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-token\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512328 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-data\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-envoy\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istiod-ca-cert\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-podinfo\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-credential-socket\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512909 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-certs\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.512909 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.512824 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-socket\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.513071 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.513039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-data\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.513193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.513178 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istiod-ca-cert\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.514763 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.514738 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-envoy\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.515003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.514986 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-podinfo\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.519769 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.519744 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-token\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.520175 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.520148 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjpcp\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-kube-api-access-mjpcp\") pod \"e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.588002 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.587966 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:18.863780 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.863753 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc"] Sep 10 14:37:18.866616 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:37:18.866589 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd84c3e_4260_4484_b5e0_906a26b4fea6.slice/crio-92a89fbd49d3ac6f29d4201caa44a1117661706eaa25839c0f0bf24e3f49b0c7 WatchSource:0}: Error finding container 92a89fbd49d3ac6f29d4201caa44a1117661706eaa25839c0f0bf24e3f49b0c7: Status 404 returned error can't find the container with id 92a89fbd49d3ac6f29d4201caa44a1117661706eaa25839c0f0bf24e3f49b0c7 Sep 10 14:37:18.869553 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.869509 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:37:18.869632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.869600 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:37:18.869685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:18.869668 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:37:19.798388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:19.798353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" event={"ID":"2cd84c3e-4260-4484-b5e0-906a26b4fea6","Type":"ContainerStarted","Data":"b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09"} Sep 10 14:37:19.798768 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:19.798408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" event={"ID":"2cd84c3e-4260-4484-b5e0-906a26b4fea6","Type":"ContainerStarted","Data":"92a89fbd49d3ac6f29d4201caa44a1117661706eaa25839c0f0bf24e3f49b0c7"} Sep 10 14:37:19.822306 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:19.822257 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podStartSLOduration=1.822243619 podStartE2EDuration="1.822243619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:19.821149667 +0000 UTC m=+1057.675058343" watchObservedRunningTime="2026-09-10 14:37:19.822243619 +0000 UTC m=+1057.676152294" Sep 10 14:37:20.588525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:20.588486 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:20.593763 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:20.593735 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:20.803411 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:20.803377 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:20.804334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:20.804306 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:21.809194 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:21.809154 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:21.901777 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:21.901739 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:22.932282 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:22.932251 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:23.033023 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.032991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:23.395521 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.395466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:23.510862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.510821 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:23.604509 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.604470 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:23.612100 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.612072 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs"] Sep 10 14:37:23.615325 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.615308 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.618124 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.618103 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-fc06d4\"" Sep 10 14:37:23.618251 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.618168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-fc06d4-dockercfg-hqxl6\"" Sep 10 14:37:23.627333 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.627311 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs"] Sep 10 14:37:23.640393 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.640372 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn"] Sep 10 14:37:23.643626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.643602 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.655774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.655748 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn"] Sep 10 14:37:23.762336 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.762301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgddb\" (UniqueName: \"kubernetes.io/projected/e387962c-7a02-47de-9312-2efb6e7e57d3-kube-api-access-tgddb\") pod \"payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.762509 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.762370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44pgt\" (UniqueName: \"kubernetes.io/projected/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-kube-api-access-44pgt\") pod \"payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.762509 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.762479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e387962c-7a02-47de-9312-2efb6e7e57d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.762593 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.762517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.863186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.863148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e387962c-7a02-47de-9312-2efb6e7e57d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.863363 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.863193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.863363 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.863251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgddb\" (UniqueName: \"kubernetes.io/projected/e387962c-7a02-47de-9312-2efb6e7e57d3-kube-api-access-tgddb\") pod \"payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.863363 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.863319 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44pgt\" (UniqueName: \"kubernetes.io/projected/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-kube-api-access-44pgt\") pod \"payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.863997 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.863970 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e387962c-7a02-47de-9312-2efb6e7e57d3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.863997 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.863995 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.873973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.873941 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44pgt\" (UniqueName: \"kubernetes.io/projected/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-kube-api-access-44pgt\") pod \"payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.874083 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.874068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgddb\" (UniqueName: \"kubernetes.io/projected/e387962c-7a02-47de-9312-2efb6e7e57d3-kube-api-access-tgddb\") pod \"payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:23.893379 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.893354 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:23.926533 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.926471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:23.954977 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:23.954924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:24.207677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.207589 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs"] Sep 10 14:37:24.210232 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:37:24.210192 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6fc99d_6725_46b4_bc3e_bf95d0cd1541.slice/crio-0c41c3038a157dcd3a35da2273d81da6c089311b9a554feddf0d84d9fc5dc164 WatchSource:0}: Error finding container 0c41c3038a157dcd3a35da2273d81da6c089311b9a554feddf0d84d9fc5dc164: Status 404 returned error can't find the container with id 0c41c3038a157dcd3a35da2273d81da6c089311b9a554feddf0d84d9fc5dc164 Sep 10 14:37:24.252326 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.252297 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn"] Sep 10 14:37:24.261864 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:37:24.261835 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode387962c_7a02_47de_9312_2efb6e7e57d3.slice/crio-c0cd01e4c8044a26b83cef879735b70af44071810e16e429c1ae397b2d40a05c WatchSource:0}: Error finding container c0cd01e4c8044a26b83cef879735b70af44071810e16e429c1ae397b2d40a05c: Status 404 returned error can't find the container with id c0cd01e4c8044a26b83cef879735b70af44071810e16e429c1ae397b2d40a05c Sep 10 14:37:24.634535 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.634502 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:24.825582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.825546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" event={"ID":"e387962c-7a02-47de-9312-2efb6e7e57d3","Type":"ContainerStarted","Data":"74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305"} Sep 10 14:37:24.825779 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.825610 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" event={"ID":"e387962c-7a02-47de-9312-2efb6e7e57d3","Type":"ContainerStarted","Data":"c0cd01e4c8044a26b83cef879735b70af44071810e16e429c1ae397b2d40a05c"} Sep 10 14:37:24.825779 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.825754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:24.827186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.827159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" event={"ID":"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541","Type":"ContainerStarted","Data":"73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6"} Sep 10 14:37:24.827318 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.827206 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" event={"ID":"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541","Type":"ContainerStarted","Data":"0c41c3038a157dcd3a35da2273d81da6c089311b9a554feddf0d84d9fc5dc164"} Sep 10 14:37:24.827318 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.827234 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:24.844459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.844415 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" podStartSLOduration=1.844400949 podStartE2EDuration="1.844400949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:24.842831673 +0000 UTC m=+1062.696740376" watchObservedRunningTime="2026-09-10 14:37:24.844400949 +0000 UTC m=+1062.698309625" Sep 10 14:37:24.863257 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:24.863212 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" podStartSLOduration=1.863199463 podStartE2EDuration="1.863199463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:24.862037647 +0000 UTC m=+1062.715946315" watchObservedRunningTime="2026-09-10 14:37:24.863199463 +0000 UTC m=+1062.717108138" Sep 10 14:37:25.270337 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:25.270301 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:26.397263 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:26.397224 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:26.722356 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:26.722267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:27.028414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:27.028326 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:27.315458 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:27.315362 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:29.212428 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:29.212385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:37:29.707838 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:29.707722 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:29.806844 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:29.806809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:30.955643 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:30.955572 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:31.056240 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.056205 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:31.392290 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.392256 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:31.477311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.477274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:31.535407 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.535370 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c"] Sep 10 14:37:31.540247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.540230 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.542953 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.542932 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-59dc59\"" Sep 10 14:37:31.543065 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.542992 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-59dc59-dockercfg-4rrg5\"" Sep 10 14:37:31.550763 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.550738 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c"] Sep 10 14:37:31.562304 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.562271 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:31.734009 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.733894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12896b5b-aaca-48de-aceb-7246546660f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.734188 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.734047 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjn28\" (UniqueName: \"kubernetes.io/projected/12896b5b-aaca-48de-aceb-7246546660f8-kube-api-access-gjn28\") pod \"payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.835426 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.835380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12896b5b-aaca-48de-aceb-7246546660f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.835616 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.835447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjn28\" (UniqueName: \"kubernetes.io/projected/12896b5b-aaca-48de-aceb-7246546660f8-kube-api-access-gjn28\") pod \"payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.836131 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.836109 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12896b5b-aaca-48de-aceb-7246546660f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.843937 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.843905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjn28\" (UniqueName: \"kubernetes.io/projected/12896b5b-aaca-48de-aceb-7246546660f8-kube-api-access-gjn28\") pod \"payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:31.852391 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:31.852366 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:32.143515 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:32.143490 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c"] Sep 10 14:37:32.145531 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:37:32.145507 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12896b5b_aaca_48de_aceb_7246546660f8.slice/crio-3cbacc78428c836007932440b2a677ed827e7d6e0563f58dcad5b408d663901f WatchSource:0}: Error finding container 3cbacc78428c836007932440b2a677ed827e7d6e0563f58dcad5b408d663901f: Status 404 returned error can't find the container with id 3cbacc78428c836007932440b2a677ed827e7d6e0563f58dcad5b408d663901f Sep 10 14:37:32.888269 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:32.888233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" event={"ID":"12896b5b-aaca-48de-aceb-7246546660f8","Type":"ContainerStarted","Data":"9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07"} Sep 10 14:37:32.888513 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:32.888284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" event={"ID":"12896b5b-aaca-48de-aceb-7246546660f8","Type":"ContainerStarted","Data":"3cbacc78428c836007932440b2a677ed827e7d6e0563f58dcad5b408d663901f"} Sep 10 14:37:32.888513 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:32.888379 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:32.904671 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:32.904622 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" podStartSLOduration=1.904608835 podStartE2EDuration="1.904608835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:37:32.902825058 +0000 UTC m=+1070.756733757" watchObservedRunningTime="2026-09-10 14:37:32.904608835 +0000 UTC m=+1070.758517511" Sep 10 14:37:35.835053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:35.834974 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:37:35.835053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:35.835048 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:37:40.932480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:40.932446 2576 generic.go:356] "Generic (PLEG): container finished" podID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerID="b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09" exitCode=0 Sep 10 14:37:40.932915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:40.932502 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" event={"ID":"2cd84c3e-4260-4484-b5e0-906a26b4fea6","Type":"ContainerDied","Data":"b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09"} Sep 10 14:37:40.932915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:40.932877 2576 scope.go:117] "RemoveContainer" containerID="b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09" Sep 10 14:37:40.933902 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:40.933873 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:37:40.933967 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:40.933940 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:37:40.934010 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:40.933972 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Sep 10 14:37:41.939524 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:41.939491 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" event={"ID":"2cd84c3e-4260-4484-b5e0-906a26b4fea6","Type":"ContainerStarted","Data":"a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee"} Sep 10 14:37:42.588280 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:42.588242 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:42.588508 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:42.588483 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 14:37:42.588589 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:42.588540 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 14:37:43.588506 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:43.588462 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 14:37:43.589012 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:43.588529 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 14:37:43.867905 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:43.867777 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:43.895016 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:43.894979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:37:44.219945 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:44.219911 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:37:44.589000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:44.588908 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 14:37:44.589000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:44.588965 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 14:37:45.589038 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:45.589004 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 14:37:45.589452 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:45.589056 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 14:37:46.588910 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:46.588868 2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 10 14:37:46.589095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:46.588930 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 10 14:37:46.667841 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:46.667778 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:47.593014 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:47.592982 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:47.593440 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:47.593288 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:47.593930 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:47.593914 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:37:51.806682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:51.806644 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:54.647080 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:54.647041 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:55.425338 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:55.425291 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:55.697066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:55.696983 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:55.817367 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:55.817328 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:57.978601 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:57.978555 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:58.816956 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:58.816921 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:58.908997 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:58.908940 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:37:59.964781 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:37:59.964746 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:00.063399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.063356 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:00.163210 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.163173 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:00.416957 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.416922 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:00.511545 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.511509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:00.563940 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.563907 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd"] Sep 10 14:38:00.568432 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.568416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.571373 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.571354 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25792506-dockercfg-p5q4m\"" Sep 10 14:38:00.571438 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.571361 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25792506\"" Sep 10 14:38:00.579116 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.579092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd"] Sep 10 14:38:00.709201 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.709119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mdk\" (UniqueName: \"kubernetes.io/projected/3091b58e-724d-4d82-962c-79839e2a81ab-kube-api-access-p7mdk\") pod \"payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.709342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.709239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3091b58e-724d-4d82-962c-79839e2a81ab-plugins-config-volume\") pod \"payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.810486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.810450 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mdk\" (UniqueName: \"kubernetes.io/projected/3091b58e-724d-4d82-962c-79839e2a81ab-kube-api-access-p7mdk\") pod \"payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.810657 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.810544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3091b58e-724d-4d82-962c-79839e2a81ab-plugins-config-volume\") pod \"payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.811237 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.811215 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3091b58e-724d-4d82-962c-79839e2a81ab-plugins-config-volume\") pod \"payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.818782 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.818756 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mdk\" (UniqueName: \"kubernetes.io/projected/3091b58e-724d-4d82-962c-79839e2a81ab-kube-api-access-p7mdk\") pod \"payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:00.879313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:00.879284 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:01.150016 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:01.149992 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd"] Sep 10 14:38:01.151578 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:38:01.151551 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3091b58e_724d_4d82_962c_79839e2a81ab.slice/crio-8efab084e84d6e1de6a6dfef82f0d9ba7db38881642343542c0fabcea4cf3e61 WatchSource:0}: Error finding container 8efab084e84d6e1de6a6dfef82f0d9ba7db38881642343542c0fabcea4cf3e61: Status 404 returned error can't find the container with id 8efab084e84d6e1de6a6dfef82f0d9ba7db38881642343542c0fabcea4cf3e61 Sep 10 14:38:02.048285 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:02.048250 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" event={"ID":"3091b58e-724d-4d82-962c-79839e2a81ab","Type":"ContainerStarted","Data":"a996537ca9683c6efbd0ae084d2df7f3abc6b06968e33424e0852c1ed1962ec9"} Sep 10 14:38:02.048561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:02.048300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" event={"ID":"3091b58e-724d-4d82-962c-79839e2a81ab","Type":"ContainerStarted","Data":"8efab084e84d6e1de6a6dfef82f0d9ba7db38881642343542c0fabcea4cf3e61"} Sep 10 14:38:02.048561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:02.048338 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:02.065862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:02.065785 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" podStartSLOduration=2.065769602 podStartE2EDuration="2.065769602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:02.064739183 +0000 UTC m=+1099.918647859" watchObservedRunningTime="2026-09-10 14:38:02.065769602 +0000 UTC m=+1099.919678317" Sep 10 14:38:07.810168 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:07.810134 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:38:11.044531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:11.044461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:13.056334 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:13.056299 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:38:23.301461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:23.301423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:38:23.778629 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:23.778590 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:25.698267 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:25.698233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:25.751718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:25.751686 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:41.542380 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:41.542344 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:42.696094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:42.696065 2576 scope.go:117] "RemoveContainer" containerID="e7ff4203e3719e3e4067e35f2defea7e4edc1734f14b273c0b8b08b049c60646" Sep 10 14:38:46.624645 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:46.624607 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:38:57.607867 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.607832 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w"] Sep 10 14:38:57.612952 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.612935 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.617007 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.616985 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"openshift-service-ca.crt\"" Sep 10 14:38:57.617113 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.617029 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"kube-root-ca.crt\"" Sep 10 14:38:57.617113 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.617047 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"auth-test-model-59dc59-kserve-self-signed-certs\"" Sep 10 14:38:57.617215 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.617047 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-59dc59\"/\"default-dockercfg-kcnjl\"" Sep 10 14:38:57.622016 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.621996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w"] Sep 10 14:38:57.627335 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.627315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-dshm\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.627441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.627348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njmmf\" (UniqueName: \"kubernetes.io/projected/09ee3f2b-02bd-4e49-b66b-279b40d1d953-kube-api-access-njmmf\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.627441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.627368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tls-certs\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.627441 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.627385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-model-cache\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.627614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.627462 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-home\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.627614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.627518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tmp-dir\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.637484 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.637439 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:38:57.728532 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.728484 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-dshm\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.728745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.728541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njmmf\" (UniqueName: \"kubernetes.io/projected/09ee3f2b-02bd-4e49-b66b-279b40d1d953-kube-api-access-njmmf\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.728745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.728570 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tls-certs\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.728745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.728593 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-model-cache\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.728745 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.728627 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-home\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.729032 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.728777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tmp-dir\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.729095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.729046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-home\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.729095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.729073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-model-cache\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.729095 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.729087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tmp-dir\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.730847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.730818 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-dshm\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.731078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.731057 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tls-certs\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.738123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.738093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njmmf\" (UniqueName: \"kubernetes.io/projected/09ee3f2b-02bd-4e49-b66b-279b40d1d953-kube-api-access-njmmf\") pod \"auth-test-model-59dc59-kserve-64fb6d7875-b546w\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:57.922948 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:57.922914 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:38:58.218064 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:58.218037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w"] Sep 10 14:38:58.220225 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:38:58.220200 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09ee3f2b_02bd_4e49_b66b_279b40d1d953.slice/crio-bbd2d478373d9599a4e38c04ce942b560cf6757e18a0fefe21df0870bb550393 WatchSource:0}: Error finding container bbd2d478373d9599a4e38c04ce942b560cf6757e18a0fefe21df0870bb550393: Status 404 returned error can't find the container with id bbd2d478373d9599a4e38c04ce942b560cf6757e18a0fefe21df0870bb550393 Sep 10 14:38:58.341193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:58.341159 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" event={"ID":"09ee3f2b-02bd-4e49-b66b-279b40d1d953","Type":"ContainerStarted","Data":"8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e"} Sep 10 14:38:58.341388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:58.341203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" event={"ID":"09ee3f2b-02bd-4e49-b66b-279b40d1d953","Type":"ContainerStarted","Data":"bbd2d478373d9599a4e38c04ce942b560cf6757e18a0fefe21df0870bb550393"} Sep 10 14:38:58.366313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:38:58.366245 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" podStartSLOduration=1.366223764 podStartE2EDuration="1.366223764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:38:58.36415671 +0000 UTC m=+1156.218065386" watchObservedRunningTime="2026-09-10 14:38:58.366223764 +0000 UTC m=+1156.220132437" Sep 10 14:39:01.432880 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:01.432846 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:04.710378 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:04.710342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:39:07.923660 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:07.923617 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:39:07.924106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:07.923699 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:39:07.936191 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:07.936168 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:39:08.405379 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:08.405353 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:39:22.124531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:22.124483 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:39:29.026326 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.026291 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx"] Sep 10 14:39:29.030045 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.030024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:29.036185 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.036160 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:39:29.036299 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.036171 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:39:29.046837 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.046811 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx"] Sep 10 14:39:29.119638 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.119597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185f9b61-f6fa-4298-b9aa-12bebe357709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25792506-rrbwx\" (UID: \"185f9b61-f6fa-4298-b9aa-12bebe357709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:29.175922 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.175877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:29.220699 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.220661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185f9b61-f6fa-4298-b9aa-12bebe357709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25792506-rrbwx\" (UID: \"185f9b61-f6fa-4298-b9aa-12bebe357709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:29.221307 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.221287 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185f9b61-f6fa-4298-b9aa-12bebe357709-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-25792506-rrbwx\" (UID: \"185f9b61-f6fa-4298-b9aa-12bebe357709\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:29.340339 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.340252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:29.616356 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:29.616131 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx"] Sep 10 14:39:29.619207 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:39:29.619178 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185f9b61_f6fa_4298_b9aa_12bebe357709.slice/crio-68d5c3b4cbbdd19c4252203ea915913c91c4226a4e2618fd483104a4bf34f697 WatchSource:0}: Error finding container 68d5c3b4cbbdd19c4252203ea915913c91c4226a4e2618fd483104a4bf34f697: Status 404 returned error can't find the container with id 68d5c3b4cbbdd19c4252203ea915913c91c4226a4e2618fd483104a4bf34f697 Sep 10 14:39:30.518526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:30.518487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" event={"ID":"185f9b61-f6fa-4298-b9aa-12bebe357709","Type":"ContainerStarted","Data":"9845d2a1710de75fb9eca43b8d6ceb27be28a4e51f3eb15ca31afe82fe81bfa4"} Sep 10 14:39:30.518918 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:30.518548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" event={"ID":"185f9b61-f6fa-4298-b9aa-12bebe357709","Type":"ContainerStarted","Data":"68d5c3b4cbbdd19c4252203ea915913c91c4226a4e2618fd483104a4bf34f697"} Sep 10 14:39:30.534992 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:30.534946 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" podStartSLOduration=2.534931648 podStartE2EDuration="2.534931648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:39:30.533253516 +0000 UTC m=+1188.387162191" watchObservedRunningTime="2026-09-10 14:39:30.534931648 +0000 UTC m=+1188.388840326" Sep 10 14:39:31.525723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:31.525690 2576 generic.go:356] "Generic (PLEG): container finished" podID="185f9b61-f6fa-4298-b9aa-12bebe357709" containerID="9845d2a1710de75fb9eca43b8d6ceb27be28a4e51f3eb15ca31afe82fe81bfa4" exitCode=0 Sep 10 14:39:31.526120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:31.525782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" event={"ID":"185f9b61-f6fa-4298-b9aa-12bebe357709","Type":"ContainerDied","Data":"9845d2a1710de75fb9eca43b8d6ceb27be28a4e51f3eb15ca31afe82fe81bfa4"} Sep 10 14:39:32.660447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:32.660420 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:32.752551 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:32.752521 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185f9b61-f6fa-4298-b9aa-12bebe357709-maas-api-service-ca\") pod \"185f9b61-f6fa-4298-b9aa-12bebe357709\" (UID: \"185f9b61-f6fa-4298-b9aa-12bebe357709\") " Sep 10 14:39:32.752924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:32.752897 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185f9b61-f6fa-4298-b9aa-12bebe357709-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "185f9b61-f6fa-4298-b9aa-12bebe357709" (UID: "185f9b61-f6fa-4298-b9aa-12bebe357709"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:39:32.854271 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:32.854190 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/185f9b61-f6fa-4298-b9aa-12bebe357709-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:39:33.537628 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:33.537594 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" event={"ID":"185f9b61-f6fa-4298-b9aa-12bebe357709","Type":"ContainerDied","Data":"68d5c3b4cbbdd19c4252203ea915913c91c4226a4e2618fd483104a4bf34f697"} Sep 10 14:39:33.537778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:33.537641 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d5c3b4cbbdd19c4252203ea915913c91c4226a4e2618fd483104a4bf34f697" Sep 10 14:39:33.537778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:33.537606 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx" Sep 10 14:39:39.008840 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:39.008786 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:39:42.618103 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:42.618074 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:39:42.620555 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:42.620534 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:39:42.717165 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:42.717134 2576 scope.go:117] "RemoveContainer" containerID="a7ed09907df3dcfa5b9ea124e9616da099ecb4c83e56342fef938cbcfa17c3fb" Sep 10 14:39:42.725313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:42.725295 2576 scope.go:117] "RemoveContainer" containerID="b9da05b761f9fc57f0dfd011ab0f115c19bcbf7a565ff9185004dfbc40efc405" Sep 10 14:39:44.406682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.406639 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:44.427035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.427000 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd"] Sep 10 14:39:44.427234 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.427207 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" podUID="3091b58e-724d-4d82-962c-79839e2a81ab" containerName="payload-processing" containerID="cri-o://a996537ca9683c6efbd0ae084d2df7f3abc6b06968e33424e0852c1ed1962ec9" gracePeriod=30 Sep 10 14:39:44.510856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.510772 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:44.599087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.599057 2576 generic.go:356] "Generic (PLEG): container finished" podID="3091b58e-724d-4d82-962c-79839e2a81ab" containerID="a996537ca9683c6efbd0ae084d2df7f3abc6b06968e33424e0852c1ed1962ec9" exitCode=0 Sep 10 14:39:44.599213 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.599087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" event={"ID":"3091b58e-724d-4d82-962c-79839e2a81ab","Type":"ContainerDied","Data":"a996537ca9683c6efbd0ae084d2df7f3abc6b06968e33424e0852c1ed1962ec9"} Sep 10 14:39:44.609287 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.609249 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:44.633156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.633130 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx"] Sep 10 14:39:44.636341 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.636315 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-25792506-rrbwx"] Sep 10 14:39:44.684354 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.684335 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:39:44.708399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.708371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:44.761354 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.761329 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="185f9b61-f6fa-4298-b9aa-12bebe357709" path="/var/lib/kubelet/pods/185f9b61-f6fa-4298-b9aa-12bebe357709/volumes" Sep 10 14:39:44.761670 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.761649 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7mdk\" (UniqueName: \"kubernetes.io/projected/3091b58e-724d-4d82-962c-79839e2a81ab-kube-api-access-p7mdk\") pod \"3091b58e-724d-4d82-962c-79839e2a81ab\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " Sep 10 14:39:44.761742 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.761726 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3091b58e-724d-4d82-962c-79839e2a81ab-plugins-config-volume\") pod \"3091b58e-724d-4d82-962c-79839e2a81ab\" (UID: \"3091b58e-724d-4d82-962c-79839e2a81ab\") " Sep 10 14:39:44.762087 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.762065 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3091b58e-724d-4d82-962c-79839e2a81ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3091b58e-724d-4d82-962c-79839e2a81ab" (UID: "3091b58e-724d-4d82-962c-79839e2a81ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:39:44.763827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.763786 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3091b58e-724d-4d82-962c-79839e2a81ab-kube-api-access-p7mdk" (OuterVolumeSpecName: "kube-api-access-p7mdk") pod "3091b58e-724d-4d82-962c-79839e2a81ab" (UID: "3091b58e-724d-4d82-962c-79839e2a81ab"). InnerVolumeSpecName "kube-api-access-p7mdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:39:44.863002 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.862963 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3091b58e-724d-4d82-962c-79839e2a81ab-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:39:44.863002 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:44.862998 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mdk\" (UniqueName: \"kubernetes.io/projected/3091b58e-724d-4d82-962c-79839e2a81ab-kube-api-access-p7mdk\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:39:45.605636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:45.605601 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" event={"ID":"3091b58e-724d-4d82-962c-79839e2a81ab","Type":"ContainerDied","Data":"8efab084e84d6e1de6a6dfef82f0d9ba7db38881642343542c0fabcea4cf3e61"} Sep 10 14:39:45.606060 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:45.605655 2576 scope.go:117] "RemoveContainer" containerID="a996537ca9683c6efbd0ae084d2df7f3abc6b06968e33424e0852c1ed1962ec9" Sep 10 14:39:45.606060 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:45.605613 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd" Sep 10 14:39:45.630807 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:45.629164 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd"] Sep 10 14:39:45.636361 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:45.636335 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25792506-7594f8f6bf-8djsd"] Sep 10 14:39:46.762167 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:46.762135 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3091b58e-724d-4d82-962c-79839e2a81ab" path="/var/lib/kubelet/pods/3091b58e-724d-4d82-962c-79839e2a81ab/volumes" Sep 10 14:39:47.717312 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:47.717276 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:52.800034 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:52.799988 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:53.788988 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:53.788943 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:39:56.915248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:39:56.915219 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:40:11.704360 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:11.704325 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:40:27.507072 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:27.507037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:40:30.464252 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:30.464216 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:32.295459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:32.295420 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:36.207777 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:36.207680 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:36.267870 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:36.267826 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:36.469737 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:36.469647 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:37.833519 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:37.833483 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:37.931228 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:37.931193 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:38.030942 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:38.030904 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:38.656734 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:38.656693 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:38.755374 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:38.755329 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:38.859956 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:38.859920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:40.094508 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:40.094466 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:42.746903 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:42.746869 2576 scope.go:117] "RemoveContainer" containerID="cd190a6327c3620ba8ee67362f5a073260c1df1caba968d30651f8577a12b3f6" Sep 10 14:40:43.508836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:43.508800 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:40:45.995168 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:45.995129 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:46.240074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:46.240026 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:46.715424 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:46.715375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:52.515273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:52.515233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:52.623655 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:52.623620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:40:52.718693 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:40:52.718658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:00.305132 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:00.305098 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:41:03.285184 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:03.285140 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:03.460973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:03.460935 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:05.318937 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:05.318902 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:06.215632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:06.215598 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w"] Sep 10 14:41:06.263474 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:06.263442 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:12.322433 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:12.322396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:12.431597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:12.431565 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:15.405711 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:15.405674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:41:18.444559 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:18.444519 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:21.235604 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.235544 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" podUID="09ee3f2b-02bd-4e49-b66b-279b40d1d953" containerName="main" containerID="cri-o://8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e" gracePeriod=45 Sep 10 14:41:21.491312 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.491253 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:41:21.537172 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tmp-dir\") pod \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " Sep 10 14:41:21.537300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537181 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tls-certs\") pod \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " Sep 10 14:41:21.537300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537208 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njmmf\" (UniqueName: \"kubernetes.io/projected/09ee3f2b-02bd-4e49-b66b-279b40d1d953-kube-api-access-njmmf\") pod \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " Sep 10 14:41:21.537300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537288 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-dshm\") pod \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " Sep 10 14:41:21.537447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537409 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-model-cache\") pod \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " Sep 10 14:41:21.537507 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537441 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09ee3f2b-02bd-4e49-b66b-279b40d1d953" (UID: "09ee3f2b-02bd-4e49-b66b-279b40d1d953"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:41:21.537507 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537458 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-home\") pod \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\" (UID: \"09ee3f2b-02bd-4e49-b66b-279b40d1d953\") " Sep 10 14:41:21.537723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537668 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-model-cache" (OuterVolumeSpecName: "model-cache") pod "09ee3f2b-02bd-4e49-b66b-279b40d1d953" (UID: "09ee3f2b-02bd-4e49-b66b-279b40d1d953"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:41:21.537879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537726 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-home" (OuterVolumeSpecName: "home") pod "09ee3f2b-02bd-4e49-b66b-279b40d1d953" (UID: "09ee3f2b-02bd-4e49-b66b-279b40d1d953"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:41:21.537879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537854 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tmp-dir\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:41:21.537879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537876 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-model-cache\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:41:21.538066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.537889 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-home\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:41:21.539349 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.539315 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-dshm" (OuterVolumeSpecName: "dshm") pod "09ee3f2b-02bd-4e49-b66b-279b40d1d953" (UID: "09ee3f2b-02bd-4e49-b66b-279b40d1d953"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:41:21.539437 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.539388 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09ee3f2b-02bd-4e49-b66b-279b40d1d953" (UID: "09ee3f2b-02bd-4e49-b66b-279b40d1d953"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 14:41:21.539437 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.539400 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ee3f2b-02bd-4e49-b66b-279b40d1d953-kube-api-access-njmmf" (OuterVolumeSpecName: "kube-api-access-njmmf") pod "09ee3f2b-02bd-4e49-b66b-279b40d1d953" (UID: "09ee3f2b-02bd-4e49-b66b-279b40d1d953"). InnerVolumeSpecName "kube-api-access-njmmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:41:21.638704 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.638669 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ee3f2b-02bd-4e49-b66b-279b40d1d953-dshm\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:41:21.638704 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.638698 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ee3f2b-02bd-4e49-b66b-279b40d1d953-tls-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:41:21.638704 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:21.638710 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njmmf\" (UniqueName: \"kubernetes.io/projected/09ee3f2b-02bd-4e49-b66b-279b40d1d953-kube-api-access-njmmf\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:41:22.085182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.085146 2576 generic.go:356] "Generic (PLEG): container finished" podID="09ee3f2b-02bd-4e49-b66b-279b40d1d953" containerID="8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e" exitCode=0 Sep 10 14:41:22.085345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.085205 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" Sep 10 14:41:22.085345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.085233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" event={"ID":"09ee3f2b-02bd-4e49-b66b-279b40d1d953","Type":"ContainerDied","Data":"8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e"} Sep 10 14:41:22.085345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.085278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w" event={"ID":"09ee3f2b-02bd-4e49-b66b-279b40d1d953","Type":"ContainerDied","Data":"bbd2d478373d9599a4e38c04ce942b560cf6757e18a0fefe21df0870bb550393"} Sep 10 14:41:22.085345 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.085297 2576 scope.go:117] "RemoveContainer" containerID="8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e" Sep 10 14:41:22.094074 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.094052 2576 scope.go:117] "RemoveContainer" containerID="8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e" Sep 10 14:41:22.094382 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:41:22.094360 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e\": container with ID starting with 8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e not found: ID does not exist" containerID="8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e" Sep 10 14:41:22.094467 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.094394 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e"} err="failed to get container status \"8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e\": rpc error: code = NotFound desc = could not find container \"8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e\": container with ID starting with 8b8e98b30bad9cd4b401578a9e911c693d468dd1d9d0a02800e898a3e7da252e not found: ID does not exist" Sep 10 14:41:22.110429 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.110395 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w"] Sep 10 14:41:22.114778 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.114752 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-59dc59/auth-test-model-59dc59-kserve-64fb6d7875-b546w"] Sep 10 14:41:22.767586 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:22.767550 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ee3f2b-02bd-4e49-b66b-279b40d1d953" path="/var/lib/kubelet/pods/09ee3f2b-02bd-4e49-b66b-279b40d1d953/volumes" Sep 10 14:41:24.410423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:24.410384 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:29.410135 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:29.410098 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:41:30.796130 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:30.796099 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:30.903660 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:30.903620 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:31.159915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:31.159861 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:31.504449 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:31.504360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:34.089201 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:34.089159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:34.183544 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:34.183512 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:35.342809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.342753 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:35.443361 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.443321 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:35.829702 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.829662 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:35.925516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.925475 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:35.970168 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970141 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x"] Sep 10 14:41:35.970513 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970501 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ee3f2b-02bd-4e49-b66b-279b40d1d953" containerName="main" Sep 10 14:41:35.970560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970514 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ee3f2b-02bd-4e49-b66b-279b40d1d953" containerName="main" Sep 10 14:41:35.970560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970527 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3091b58e-724d-4d82-962c-79839e2a81ab" containerName="payload-processing" Sep 10 14:41:35.970560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970532 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3091b58e-724d-4d82-962c-79839e2a81ab" containerName="payload-processing" Sep 10 14:41:35.970560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970539 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="185f9b61-f6fa-4298-b9aa-12bebe357709" containerName="revoke-keys" Sep 10 14:41:35.970560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970545 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="185f9b61-f6fa-4298-b9aa-12bebe357709" containerName="revoke-keys" Sep 10 14:41:35.970717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970613 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3091b58e-724d-4d82-962c-79839e2a81ab" containerName="payload-processing" Sep 10 14:41:35.970717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970621 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ee3f2b-02bd-4e49-b66b-279b40d1d953" containerName="main" Sep 10 14:41:35.970717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.970629 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="185f9b61-f6fa-4298-b9aa-12bebe357709" containerName="revoke-keys" Sep 10 14:41:35.974963 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.974946 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:35.978547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.978526 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1d2af2-dockercfg-xmrjw\"" Sep 10 14:41:35.978676 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.978534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1d2af2\"" Sep 10 14:41:35.984456 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:35.984428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x"] Sep 10 14:41:36.029606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.029568 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:36.063459 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.063425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvcfh\" (UniqueName: \"kubernetes.io/projected/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-kube-api-access-nvcfh\") pod \"payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.063637 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.063598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.165000 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.164962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvcfh\" (UniqueName: \"kubernetes.io/projected/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-kube-api-access-nvcfh\") pod \"payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.165250 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.165062 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.165772 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.165753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.178203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.178170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvcfh\" (UniqueName: \"kubernetes.io/projected/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-kube-api-access-nvcfh\") pod \"payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.285557 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.285525 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:36.556881 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.556859 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x"] Sep 10 14:41:36.559282 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:41:36.559252 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82909f9_16f4_4329_85dc_a8e8a5cbfdca.slice/crio-c979d5e126b4c505e5ad704d784a6cadcf75f7526beff09efb5f84c7415ec616 WatchSource:0}: Error finding container c979d5e126b4c505e5ad704d784a6cadcf75f7526beff09efb5f84c7415ec616: Status 404 returned error can't find the container with id c979d5e126b4c505e5ad704d784a6cadcf75f7526beff09efb5f84c7415ec616 Sep 10 14:41:36.773244 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:36.773207 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:37.163557 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:37.163514 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" event={"ID":"e82909f9-16f4-4329-85dc-a8e8a5cbfdca","Type":"ContainerStarted","Data":"b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7"} Sep 10 14:41:37.163723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:37.163574 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" event={"ID":"e82909f9-16f4-4329-85dc-a8e8a5cbfdca","Type":"ContainerStarted","Data":"c979d5e126b4c505e5ad704d784a6cadcf75f7526beff09efb5f84c7415ec616"} Sep 10 14:41:37.163723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:37.163605 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:37.182687 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:37.182637 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" podStartSLOduration=2.182620829 podStartE2EDuration="2.182620829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:41:37.181206629 +0000 UTC m=+1315.035115304" watchObservedRunningTime="2026-09-10 14:41:37.182620829 +0000 UTC m=+1315.036529505" Sep 10 14:41:42.771340 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:42.771307 2576 scope.go:117] "RemoveContainer" containerID="e3e696b3a80e957ed5d09965cccb102adac15b402d9b06a6c73fb2177160bb9c" Sep 10 14:41:44.013677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:44.013636 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:41:45.034109 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:45.034075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:48.170237 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:48.170203 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:41:57.507419 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:57.507384 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:41:59.076521 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:59.076479 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:59.613898 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:59.613859 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:41:59.942343 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:41:59.942302 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:00.351360 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:00.351264 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:03.403193 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:03.403149 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:03.503142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:03.503094 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:04.549199 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:04.549159 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:04.645536 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:04.645496 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:04.802947 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:04.802850 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:05.190306 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.190267 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:05.291301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.291265 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7"] Sep 10 14:42:05.292225 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.292194 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:05.296290 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.296272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.299097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.299075 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4ba551\"" Sep 10 14:42:05.299186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.299167 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4ba551-dockercfg-ttfqr\"" Sep 10 14:42:05.306923 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.306897 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7"] Sep 10 14:42:05.330003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.329975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7js\" (UniqueName: \"kubernetes.io/projected/251cee22-7fab-4ed4-a417-d9a0bab27ef2-kube-api-access-zh7js\") pod \"payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.330115 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.330036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251cee22-7fab-4ed4-a417-d9a0bab27ef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.430973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.430925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7js\" (UniqueName: \"kubernetes.io/projected/251cee22-7fab-4ed4-a417-d9a0bab27ef2-kube-api-access-zh7js\") pod \"payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.431140 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.431001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251cee22-7fab-4ed4-a417-d9a0bab27ef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.431641 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.431623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251cee22-7fab-4ed4-a417-d9a0bab27ef2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.438851 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.438783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7js\" (UniqueName: \"kubernetes.io/projected/251cee22-7fab-4ed4-a417-d9a0bab27ef2-kube-api-access-zh7js\") pod \"payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.607466 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.607386 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:05.878842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:05.878816 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7"] Sep 10 14:42:05.880835 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:42:05.880806 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251cee22_7fab_4ed4_a417_d9a0bab27ef2.slice/crio-44a874202b784e862aff3e70543aec48d70fab67db70e0d735ef3497368ac528 WatchSource:0}: Error finding container 44a874202b784e862aff3e70543aec48d70fab67db70e0d735ef3497368ac528: Status 404 returned error can't find the container with id 44a874202b784e862aff3e70543aec48d70fab67db70e0d735ef3497368ac528 Sep 10 14:42:06.306718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:06.306633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" event={"ID":"251cee22-7fab-4ed4-a417-d9a0bab27ef2","Type":"ContainerStarted","Data":"d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c"} Sep 10 14:42:06.306718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:06.306696 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" event={"ID":"251cee22-7fab-4ed4-a417-d9a0bab27ef2","Type":"ContainerStarted","Data":"44a874202b784e862aff3e70543aec48d70fab67db70e0d735ef3497368ac528"} Sep 10 14:42:06.306915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:06.306842 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:06.323106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:06.323060 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" podStartSLOduration=1.323046782 podStartE2EDuration="1.323046782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:06.322006 +0000 UTC m=+1344.175914678" watchObservedRunningTime="2026-09-10 14:42:06.323046782 +0000 UTC m=+1344.176955458" Sep 10 14:42:15.405962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:15.405917 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:16.314040 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:16.313999 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:42:17.314134 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:17.314104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:42:18.902388 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:18.902346 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:24.627149 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:24.627115 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:26.395809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:26.395765 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:26.456921 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:26.456885 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:27.447525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:27.447485 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:28.336842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:28.336769 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:31.718431 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.718394 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq"] Sep 10 14:42:31.722035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.722017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:31.724879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.724853 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:42:31.724997 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.724856 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:42:31.729155 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.729128 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq"] Sep 10 14:42:31.823483 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.823436 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:31.878518 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.878487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e896b9-d6ca-4a51-bc26-02260b3004ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq\" (UID: \"41e896b9-d6ca-4a51-bc26-02260b3004ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:31.979847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.979752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e896b9-d6ca-4a51-bc26-02260b3004ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq\" (UID: \"41e896b9-d6ca-4a51-bc26-02260b3004ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:31.980373 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:31.980352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e896b9-d6ca-4a51-bc26-02260b3004ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq\" (UID: \"41e896b9-d6ca-4a51-bc26-02260b3004ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:32.013435 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:32.013408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:42:32.034423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:32.034399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:32.308229 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:32.308204 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq"] Sep 10 14:42:32.313387 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:42:32.313352 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e896b9_d6ca_4a51_bc26_02260b3004ab.slice/crio-79e7ef0b8a975698ca594409ed4e7672ce019ccc8e231c82d4124c79fd1a2f54 WatchSource:0}: Error finding container 79e7ef0b8a975698ca594409ed4e7672ce019ccc8e231c82d4124c79fd1a2f54: Status 404 returned error can't find the container with id 79e7ef0b8a975698ca594409ed4e7672ce019ccc8e231c82d4124c79fd1a2f54 Sep 10 14:42:32.447529 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:32.447496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" event={"ID":"41e896b9-d6ca-4a51-bc26-02260b3004ab","Type":"ContainerStarted","Data":"79e7ef0b8a975698ca594409ed4e7672ce019ccc8e231c82d4124c79fd1a2f54"} Sep 10 14:42:33.454632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:33.454597 2576 generic.go:356] "Generic (PLEG): container finished" podID="41e896b9-d6ca-4a51-bc26-02260b3004ab" containerID="90996934b56568eb0790e5223ae1138cdb85c439184c89c056c16f1468bebaa2" exitCode=0 Sep 10 14:42:33.455141 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:33.454663 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" event={"ID":"41e896b9-d6ca-4a51-bc26-02260b3004ab","Type":"ContainerDied","Data":"90996934b56568eb0790e5223ae1138cdb85c439184c89c056c16f1468bebaa2"} Sep 10 14:42:34.594007 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:34.593985 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:34.602535 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:34.602515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e896b9-d6ca-4a51-bc26-02260b3004ab-maas-api-service-ca\") pod \"41e896b9-d6ca-4a51-bc26-02260b3004ab\" (UID: \"41e896b9-d6ca-4a51-bc26-02260b3004ab\") " Sep 10 14:42:34.602859 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:34.602838 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41e896b9-d6ca-4a51-bc26-02260b3004ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41e896b9-d6ca-4a51-bc26-02260b3004ab" (UID: "41e896b9-d6ca-4a51-bc26-02260b3004ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:42:34.703401 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:34.703370 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41e896b9-d6ca-4a51-bc26-02260b3004ab-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:42:35.469134 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:35.469102 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" Sep 10 14:42:35.469134 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:35.469125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq" event={"ID":"41e896b9-d6ca-4a51-bc26-02260b3004ab","Type":"ContainerDied","Data":"79e7ef0b8a975698ca594409ed4e7672ce019ccc8e231c82d4124c79fd1a2f54"} Sep 10 14:42:35.469337 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:35.469166 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e7ef0b8a975698ca594409ed4e7672ce019ccc8e231c82d4124c79fd1a2f54" Sep 10 14:42:38.139182 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:38.139144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:44.489696 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:44.489658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:47.169119 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.169075 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:47.198238 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.198208 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x"] Sep 10 14:42:47.198442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.198413 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" podUID="e82909f9-16f4-4329-85dc-a8e8a5cbfdca" containerName="payload-processing" containerID="cri-o://b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7" gracePeriod=30 Sep 10 14:42:47.271771 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.271733 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:47.372747 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.372556 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:47.401842 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.401780 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq"] Sep 10 14:42:47.403828 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.403772 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1d2af2-78nwq"] Sep 10 14:42:47.459677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.459647 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:42:47.469534 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.469506 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:47.506973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.506943 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvcfh\" (UniqueName: \"kubernetes.io/projected/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-kube-api-access-nvcfh\") pod \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " Sep 10 14:42:47.507122 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.506980 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-plugins-config-volume\") pod \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\" (UID: \"e82909f9-16f4-4329-85dc-a8e8a5cbfdca\") " Sep 10 14:42:47.507355 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.507327 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e82909f9-16f4-4329-85dc-a8e8a5cbfdca" (UID: "e82909f9-16f4-4329-85dc-a8e8a5cbfdca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:42:47.509042 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.509015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-kube-api-access-nvcfh" (OuterVolumeSpecName: "kube-api-access-nvcfh") pod "e82909f9-16f4-4329-85dc-a8e8a5cbfdca" (UID: "e82909f9-16f4-4329-85dc-a8e8a5cbfdca"). InnerVolumeSpecName "kube-api-access-nvcfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:42:47.531784 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.531757 2576 generic.go:356] "Generic (PLEG): container finished" podID="e82909f9-16f4-4329-85dc-a8e8a5cbfdca" containerID="b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7" exitCode=0 Sep 10 14:42:47.531938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.531826 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" Sep 10 14:42:47.531938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.531830 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" event={"ID":"e82909f9-16f4-4329-85dc-a8e8a5cbfdca","Type":"ContainerDied","Data":"b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7"} Sep 10 14:42:47.532018 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.531953 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x" event={"ID":"e82909f9-16f4-4329-85dc-a8e8a5cbfdca","Type":"ContainerDied","Data":"c979d5e126b4c505e5ad704d784a6cadcf75f7526beff09efb5f84c7415ec616"} Sep 10 14:42:47.532018 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.531980 2576 scope.go:117] "RemoveContainer" containerID="b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7" Sep 10 14:42:47.540457 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.540439 2576 scope.go:117] "RemoveContainer" containerID="b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7" Sep 10 14:42:47.540732 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:42:47.540713 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7\": container with ID starting with b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7 not found: ID does not exist" containerID="b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7" Sep 10 14:42:47.540827 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.540741 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7"} err="failed to get container status \"b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7\": rpc error: code = NotFound desc = could not find container \"b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7\": container with ID starting with b18162a0eb38569cde19a89f37c9e35bfdabef2f9a440e0060568b4a466240d7 not found: ID does not exist" Sep 10 14:42:47.557836 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.557775 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x"] Sep 10 14:42:47.561156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.561133 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1d2af2-577cd885b7-8tl5x"] Sep 10 14:42:47.608640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.608611 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvcfh\" (UniqueName: \"kubernetes.io/projected/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-kube-api-access-nvcfh\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:42:47.608640 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.608637 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e82909f9-16f4-4329-85dc-a8e8a5cbfdca-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:42:47.779166 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:47.779079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:48.762485 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:48.762453 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41e896b9-d6ca-4a51-bc26-02260b3004ab" path="/var/lib/kubelet/pods/41e896b9-d6ca-4a51-bc26-02260b3004ab/volumes" Sep 10 14:42:48.762844 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:48.762829 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e82909f9-16f4-4329-85dc-a8e8a5cbfdca" path="/var/lib/kubelet/pods/e82909f9-16f4-4329-85dc-a8e8a5cbfdca/volumes" Sep 10 14:42:52.414655 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:52.414618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:52.726450 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:52.726373 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:42:53.834097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:53.834039 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:53.899686 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:53.899657 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:54.104895 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.104809 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:54.430934 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.430903 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc"] Sep 10 14:42:54.431365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.431351 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41e896b9-d6ca-4a51-bc26-02260b3004ab" containerName="revoke-keys" Sep 10 14:42:54.431425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.431368 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="41e896b9-d6ca-4a51-bc26-02260b3004ab" containerName="revoke-keys" Sep 10 14:42:54.431425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.431380 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e82909f9-16f4-4329-85dc-a8e8a5cbfdca" containerName="payload-processing" Sep 10 14:42:54.431425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.431385 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e82909f9-16f4-4329-85dc-a8e8a5cbfdca" containerName="payload-processing" Sep 10 14:42:54.431530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.431450 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="41e896b9-d6ca-4a51-bc26-02260b3004ab" containerName="revoke-keys" Sep 10 14:42:54.431530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.431463 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e82909f9-16f4-4329-85dc-a8e8a5cbfdca" containerName="payload-processing" Sep 10 14:42:54.436203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.436180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:42:54.438845 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.438820 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:42:54.439037 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.439013 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:42:54.442414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.442392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc"] Sep 10 14:42:54.468907 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.468879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128fadc9-d2ea-436d-9a00-c113cffe19cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc\" (UID: \"128fadc9-d2ea-436d-9a00-c113cffe19cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:42:54.556461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.556422 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:42:54.569782 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.569753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128fadc9-d2ea-436d-9a00-c113cffe19cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc\" (UID: \"128fadc9-d2ea-436d-9a00-c113cffe19cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:42:54.570368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.570350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128fadc9-d2ea-436d-9a00-c113cffe19cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc\" (UID: \"128fadc9-d2ea-436d-9a00-c113cffe19cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:42:54.747716 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:54.747634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:42:55.022175 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:55.022150 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc"] Sep 10 14:42:55.024197 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:42:55.024169 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128fadc9_d2ea_436d_9a00_c113cffe19cb.slice/crio-539fbf3dc064b92a4e3110c1b2ba087961584ade5a5a337c8ef16bba36e6f1e3 WatchSource:0}: Error finding container 539fbf3dc064b92a4e3110c1b2ba087961584ade5a5a337c8ef16bba36e6f1e3: Status 404 returned error can't find the container with id 539fbf3dc064b92a4e3110c1b2ba087961584ade5a5a337c8ef16bba36e6f1e3 Sep 10 14:42:55.575078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:55.574991 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" event={"ID":"128fadc9-d2ea-436d-9a00-c113cffe19cb","Type":"ContainerStarted","Data":"ab90703466e16a4a9dbca2c5a384e6168d5c68caaec72300fc13199007c27b02"} Sep 10 14:42:55.575078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:55.575046 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" event={"ID":"128fadc9-d2ea-436d-9a00-c113cffe19cb","Type":"ContainerStarted","Data":"539fbf3dc064b92a4e3110c1b2ba087961584ade5a5a337c8ef16bba36e6f1e3"} Sep 10 14:42:55.590925 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:55.590881 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" podStartSLOduration=1.590867318 podStartE2EDuration="1.590867318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:42:55.589372419 +0000 UTC m=+1393.443281095" watchObservedRunningTime="2026-09-10 14:42:55.590867318 +0000 UTC m=+1393.444775994" Sep 10 14:42:56.581152 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:56.581109 2576 generic.go:356] "Generic (PLEG): container finished" podID="128fadc9-d2ea-436d-9a00-c113cffe19cb" containerID="ab90703466e16a4a9dbca2c5a384e6168d5c68caaec72300fc13199007c27b02" exitCode=0 Sep 10 14:42:56.581518 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:56.581200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" event={"ID":"128fadc9-d2ea-436d-9a00-c113cffe19cb","Type":"ContainerDied","Data":"ab90703466e16a4a9dbca2c5a384e6168d5c68caaec72300fc13199007c27b02"} Sep 10 14:42:57.719185 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:57.719153 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:42:57.801834 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:57.801779 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128fadc9-d2ea-436d-9a00-c113cffe19cb-maas-api-service-ca\") pod \"128fadc9-d2ea-436d-9a00-c113cffe19cb\" (UID: \"128fadc9-d2ea-436d-9a00-c113cffe19cb\") " Sep 10 14:42:57.802142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:57.802121 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128fadc9-d2ea-436d-9a00-c113cffe19cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "128fadc9-d2ea-436d-9a00-c113cffe19cb" (UID: "128fadc9-d2ea-436d-9a00-c113cffe19cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:42:57.902549 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:57.902514 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/128fadc9-d2ea-436d-9a00-c113cffe19cb-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:42:58.592242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:58.592208 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" event={"ID":"128fadc9-d2ea-436d-9a00-c113cffe19cb","Type":"ContainerDied","Data":"539fbf3dc064b92a4e3110c1b2ba087961584ade5a5a337c8ef16bba36e6f1e3"} Sep 10 14:42:58.592424 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:58.592261 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="539fbf3dc064b92a4e3110c1b2ba087961584ade5a5a337c8ef16bba36e6f1e3" Sep 10 14:42:58.592424 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:42:58.592219 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc" Sep 10 14:43:09.864918 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:09.864879 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:09.904481 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:09.904449 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7"] Sep 10 14:43:09.904708 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:09.904684 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" podUID="251cee22-7fab-4ed4-a417-d9a0bab27ef2" containerName="payload-pre-processing" containerID="cri-o://d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c" gracePeriod=30 Sep 10 14:43:09.941833 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:09.941768 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:10.049650 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.049616 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:10.059053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.059026 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc"] Sep 10 14:43:10.061921 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.061897 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4ba551-4fhsc"] Sep 10 14:43:10.128053 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.127999 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:10.169758 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.169734 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:43:10.318499 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.318465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zh7js\" (UniqueName: \"kubernetes.io/projected/251cee22-7fab-4ed4-a417-d9a0bab27ef2-kube-api-access-zh7js\") pod \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " Sep 10 14:43:10.318694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.318540 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251cee22-7fab-4ed4-a417-d9a0bab27ef2-plugins-config-volume\") pod \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\" (UID: \"251cee22-7fab-4ed4-a417-d9a0bab27ef2\") " Sep 10 14:43:10.318962 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.318936 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251cee22-7fab-4ed4-a417-d9a0bab27ef2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "251cee22-7fab-4ed4-a417-d9a0bab27ef2" (UID: "251cee22-7fab-4ed4-a417-d9a0bab27ef2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:43:10.320634 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.320614 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251cee22-7fab-4ed4-a417-d9a0bab27ef2-kube-api-access-zh7js" (OuterVolumeSpecName: "kube-api-access-zh7js") pod "251cee22-7fab-4ed4-a417-d9a0bab27ef2" (UID: "251cee22-7fab-4ed4-a417-d9a0bab27ef2"). InnerVolumeSpecName "kube-api-access-zh7js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:43:10.420068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.419995 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zh7js\" (UniqueName: \"kubernetes.io/projected/251cee22-7fab-4ed4-a417-d9a0bab27ef2-kube-api-access-zh7js\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:10.420068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.420027 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251cee22-7fab-4ed4-a417-d9a0bab27ef2-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:10.434455 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.434425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:10.652606 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.652567 2576 generic.go:356] "Generic (PLEG): container finished" podID="251cee22-7fab-4ed4-a417-d9a0bab27ef2" containerID="d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c" exitCode=0 Sep 10 14:43:10.652756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.652604 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" event={"ID":"251cee22-7fab-4ed4-a417-d9a0bab27ef2","Type":"ContainerDied","Data":"d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c"} Sep 10 14:43:10.652756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.652612 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" Sep 10 14:43:10.652756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.652660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7" event={"ID":"251cee22-7fab-4ed4-a417-d9a0bab27ef2","Type":"ContainerDied","Data":"44a874202b784e862aff3e70543aec48d70fab67db70e0d735ef3497368ac528"} Sep 10 14:43:10.652756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.652683 2576 scope.go:117] "RemoveContainer" containerID="d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c" Sep 10 14:43:10.662980 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.662843 2576 scope.go:117] "RemoveContainer" containerID="d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c" Sep 10 14:43:10.663263 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:43:10.663229 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c\": container with ID starting with d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c not found: ID does not exist" containerID="d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c" Sep 10 14:43:10.663372 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.663295 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c"} err="failed to get container status \"d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c\": rpc error: code = NotFound desc = could not find container \"d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c\": container with ID starting with d03faebb03c4db49ed6b02544b1f83a799f89429f61fd83a22cd722038f5920c not found: ID does not exist" Sep 10 14:43:10.692581 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.692524 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7"] Sep 10 14:43:10.706224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.706202 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4ba551-848fb5d858-b7km7"] Sep 10 14:43:10.761955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.761931 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="128fadc9-d2ea-436d-9a00-c113cffe19cb" path="/var/lib/kubelet/pods/128fadc9-d2ea-436d-9a00-c113cffe19cb/volumes" Sep 10 14:43:10.762295 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:10.762282 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="251cee22-7fab-4ed4-a417-d9a0bab27ef2" path="/var/lib/kubelet/pods/251cee22-7fab-4ed4-a417-d9a0bab27ef2/volumes" Sep 10 14:43:14.309370 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:14.309329 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:43:16.326283 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.326247 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:16.388422 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.388387 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:16.589448 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.589360 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:16.967314 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967278 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x"] Sep 10 14:43:16.967679 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967667 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="251cee22-7fab-4ed4-a417-d9a0bab27ef2" containerName="payload-pre-processing" Sep 10 14:43:16.967723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967680 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="251cee22-7fab-4ed4-a417-d9a0bab27ef2" containerName="payload-pre-processing" Sep 10 14:43:16.967723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967694 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128fadc9-d2ea-436d-9a00-c113cffe19cb" containerName="revoke-keys" Sep 10 14:43:16.967723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967700 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="128fadc9-d2ea-436d-9a00-c113cffe19cb" containerName="revoke-keys" Sep 10 14:43:16.967835 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967758 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="251cee22-7fab-4ed4-a417-d9a0bab27ef2" containerName="payload-pre-processing" Sep 10 14:43:16.967835 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.967768 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="128fadc9-d2ea-436d-9a00-c113cffe19cb" containerName="revoke-keys" Sep 10 14:43:16.970936 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.970921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:16.974705 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.974683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:43:16.976347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.976326 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:43:16.987044 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:16.987023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x"] Sep 10 14:43:17.085320 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.085287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36fe7b9-9f57-4215-898b-eb764b353b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x\" (UID: \"d36fe7b9-9f57-4215-898b-eb764b353b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:17.176938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.176899 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:17.185954 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.185918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36fe7b9-9f57-4215-898b-eb764b353b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x\" (UID: \"d36fe7b9-9f57-4215-898b-eb764b353b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:17.186530 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.186508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36fe7b9-9f57-4215-898b-eb764b353b38-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x\" (UID: \"d36fe7b9-9f57-4215-898b-eb764b353b38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:17.280636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.280544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:17.551364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.551323 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x"] Sep 10 14:43:17.553670 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:43:17.553639 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36fe7b9_9f57_4215_898b_eb764b353b38.slice/crio-c76599dc240eb555d15c0e737656f8974d558e27256cf3f9e5e7cf70b97f51be WatchSource:0}: Error finding container c76599dc240eb555d15c0e737656f8974d558e27256cf3f9e5e7cf70b97f51be: Status 404 returned error can't find the container with id c76599dc240eb555d15c0e737656f8974d558e27256cf3f9e5e7cf70b97f51be Sep 10 14:43:17.693386 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:17.693346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" event={"ID":"d36fe7b9-9f57-4215-898b-eb764b353b38","Type":"ContainerStarted","Data":"c76599dc240eb555d15c0e737656f8974d558e27256cf3f9e5e7cf70b97f51be"} Sep 10 14:43:18.699413 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:18.699378 2576 generic.go:356] "Generic (PLEG): container finished" podID="d36fe7b9-9f57-4215-898b-eb764b353b38" containerID="283cb7f0a44f281d11ad75b96742b9aa9f8178912f102c19140b6400968eec0f" exitCode=0 Sep 10 14:43:18.699833 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:18.699429 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" event={"ID":"d36fe7b9-9f57-4215-898b-eb764b353b38","Type":"ContainerDied","Data":"283cb7f0a44f281d11ad75b96742b9aa9f8178912f102c19140b6400968eec0f"} Sep 10 14:43:19.839228 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:19.839204 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:19.908908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:19.908882 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36fe7b9-9f57-4215-898b-eb764b353b38-maas-api-service-ca\") pod \"d36fe7b9-9f57-4215-898b-eb764b353b38\" (UID: \"d36fe7b9-9f57-4215-898b-eb764b353b38\") " Sep 10 14:43:19.909273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:19.909254 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d36fe7b9-9f57-4215-898b-eb764b353b38-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d36fe7b9-9f57-4215-898b-eb764b353b38" (UID: "d36fe7b9-9f57-4215-898b-eb764b353b38"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:43:20.010089 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:20.010017 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d36fe7b9-9f57-4215-898b-eb764b353b38-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:20.717412 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:20.717378 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" Sep 10 14:43:20.717598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:20.717415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x" event={"ID":"d36fe7b9-9f57-4215-898b-eb764b353b38","Type":"ContainerDied","Data":"c76599dc240eb555d15c0e737656f8974d558e27256cf3f9e5e7cf70b97f51be"} Sep 10 14:43:20.717598 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:20.717499 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76599dc240eb555d15c0e737656f8974d558e27256cf3f9e5e7cf70b97f51be" Sep 10 14:43:27.493905 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:27.493867 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:27.595446 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:27.595404 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:27.710066 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:27.710032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:27.721216 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:27.721175 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x"] Sep 10 14:43:27.723205 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:27.723180 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ecc1f-f6r4x"] Sep 10 14:43:27.812834 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:27.812741 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:28.099186 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:28.099096 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:28.761474 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:28.761443 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d36fe7b9-9f57-4215-898b-eb764b353b38" path="/var/lib/kubelet/pods/d36fe7b9-9f57-4215-898b-eb764b353b38/volumes" Sep 10 14:43:31.815290 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:31.815262 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:43:33.883158 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:33.883117 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:33.937067 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:33.937032 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:34.139776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:34.139681 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:34.618860 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:34.618745 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:37.735144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:37.735102 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:43.606311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:43.606276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:43:44.932486 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:44.932444 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:44.965196 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:44.965166 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn"] Sep 10 14:43:44.965469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:44.965445 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" podUID="e387962c-7a02-47de-9312-2efb6e7e57d3" containerName="payload-processing" containerID="cri-o://74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305" gracePeriod=30 Sep 10 14:43:44.971461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:44.970776 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs"] Sep 10 14:43:44.971461 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:44.971029 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" podUID="2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" containerName="payload-pre-processing" containerID="cri-o://73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6" gracePeriod=30 Sep 10 14:43:45.004265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.004233 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:45.104221 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.104179 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:45.214879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.214782 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:45.231666 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.231641 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:43:45.251024 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.250879 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:43:45.328048 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.328006 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-plugins-config-volume\") pod \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " Sep 10 14:43:45.328243 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.328076 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgddb\" (UniqueName: \"kubernetes.io/projected/e387962c-7a02-47de-9312-2efb6e7e57d3-kube-api-access-tgddb\") pod \"e387962c-7a02-47de-9312-2efb6e7e57d3\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " Sep 10 14:43:45.328243 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.328123 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44pgt\" (UniqueName: \"kubernetes.io/projected/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-kube-api-access-44pgt\") pod \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\" (UID: \"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541\") " Sep 10 14:43:45.328243 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.328176 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e387962c-7a02-47de-9312-2efb6e7e57d3-plugins-config-volume\") pod \"e387962c-7a02-47de-9312-2efb6e7e57d3\" (UID: \"e387962c-7a02-47de-9312-2efb6e7e57d3\") " Sep 10 14:43:45.328501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.328470 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" (UID: "2a6fc99d-6725-46b4-bc3e-bf95d0cd1541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:43:45.328587 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.328519 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e387962c-7a02-47de-9312-2efb6e7e57d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e387962c-7a02-47de-9312-2efb6e7e57d3" (UID: "e387962c-7a02-47de-9312-2efb6e7e57d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:43:45.330281 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.330262 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-kube-api-access-44pgt" (OuterVolumeSpecName: "kube-api-access-44pgt") pod "2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" (UID: "2a6fc99d-6725-46b4-bc3e-bf95d0cd1541"). InnerVolumeSpecName "kube-api-access-44pgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:43:45.330336 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.330290 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e387962c-7a02-47de-9312-2efb6e7e57d3-kube-api-access-tgddb" (OuterVolumeSpecName: "kube-api-access-tgddb") pod "e387962c-7a02-47de-9312-2efb6e7e57d3" (UID: "e387962c-7a02-47de-9312-2efb6e7e57d3"). InnerVolumeSpecName "kube-api-access-tgddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:43:45.429512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.429475 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgddb\" (UniqueName: \"kubernetes.io/projected/e387962c-7a02-47de-9312-2efb6e7e57d3-kube-api-access-tgddb\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:45.429512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.429505 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44pgt\" (UniqueName: \"kubernetes.io/projected/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-kube-api-access-44pgt\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:45.429512 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.429516 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e387962c-7a02-47de-9312-2efb6e7e57d3-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:45.429750 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.429524 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:45.484767 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.484670 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:45.857233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.857137 2576 generic.go:356] "Generic (PLEG): container finished" podID="e387962c-7a02-47de-9312-2efb6e7e57d3" containerID="74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305" exitCode=0 Sep 10 14:43:45.857233 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.857184 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" Sep 10 14:43:45.857469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.857228 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" event={"ID":"e387962c-7a02-47de-9312-2efb6e7e57d3","Type":"ContainerDied","Data":"74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305"} Sep 10 14:43:45.857469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.857287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn" event={"ID":"e387962c-7a02-47de-9312-2efb6e7e57d3","Type":"ContainerDied","Data":"c0cd01e4c8044a26b83cef879735b70af44071810e16e429c1ae397b2d40a05c"} Sep 10 14:43:45.857469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.857321 2576 scope.go:117] "RemoveContainer" containerID="74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305" Sep 10 14:43:45.859177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.859140 2576 generic.go:356] "Generic (PLEG): container finished" podID="2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" containerID="73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6" exitCode=0 Sep 10 14:43:45.859177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.859169 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" Sep 10 14:43:45.859358 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.859176 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" event={"ID":"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541","Type":"ContainerDied","Data":"73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6"} Sep 10 14:43:45.859358 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.859231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs" event={"ID":"2a6fc99d-6725-46b4-bc3e-bf95d0cd1541","Type":"ContainerDied","Data":"0c41c3038a157dcd3a35da2273d81da6c089311b9a554feddf0d84d9fc5dc164"} Sep 10 14:43:45.870211 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.870184 2576 scope.go:117] "RemoveContainer" containerID="74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305" Sep 10 14:43:45.870527 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:43:45.870503 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305\": container with ID starting with 74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305 not found: ID does not exist" containerID="74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305" Sep 10 14:43:45.870605 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.870538 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305"} err="failed to get container status \"74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305\": rpc error: code = NotFound desc = could not find container \"74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305\": container with ID starting with 74cf20b1a5ed85ff9b4e56a8b49b9b72a8b087812de491fd9f5d802fc0d48305 not found: ID does not exist" Sep 10 14:43:45.870605 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.870561 2576 scope.go:117] "RemoveContainer" containerID="73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6" Sep 10 14:43:45.878600 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.878579 2576 scope.go:117] "RemoveContainer" containerID="73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6" Sep 10 14:43:45.878912 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:43:45.878893 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6\": container with ID starting with 73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6 not found: ID does not exist" containerID="73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6" Sep 10 14:43:45.878971 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.878920 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6"} err="failed to get container status \"73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6\": rpc error: code = NotFound desc = could not find container \"73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6\": container with ID starting with 73f62b6ce19705d0f37ba87745625a66c2034209ed84a2e912c76d543926a0b6 not found: ID does not exist" Sep 10 14:43:45.882585 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.882564 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs"] Sep 10 14:43:45.884187 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.884168 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-fc06d4-5c7c7b997d-6mxxs"] Sep 10 14:43:45.893669 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.893648 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn"] Sep 10 14:43:45.897703 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:45.897684 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-fc06d4-749c468585-hf8jn"] Sep 10 14:43:46.761289 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:46.761254 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" path="/var/lib/kubelet/pods/2a6fc99d-6725-46b4-bc3e-bf95d0cd1541/volumes" Sep 10 14:43:46.761749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:46.761731 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e387962c-7a02-47de-9312-2efb6e7e57d3" path="/var/lib/kubelet/pods/e387962c-7a02-47de-9312-2efb6e7e57d3/volumes" Sep 10 14:43:51.808799 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:51.808761 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:51.875868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:51.875835 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:52.065938 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:52.065840 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc"] Sep 10 14:43:52.066123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:52.066095 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" containerID="cri-o://a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee" gracePeriod=30 Sep 10 14:43:52.086967 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:52.086933 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:43:55.317433 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:55.317400 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:43:57.320845 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.320820 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:43:57.440517 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440469 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-credential-socket\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440517 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440526 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjpcp\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-kube-api-access-mjpcp\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440584 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-data\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440626 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-podinfo\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440705 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-token\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440756 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:43:57.440908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440775 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-envoy\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440852 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-socket\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.440908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-data" (OuterVolumeSpecName: "istio-data") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:43:57.440908 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440887 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-certs\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.441185 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.440925 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istiod-ca-cert\") pod \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\" (UID: \"2cd84c3e-4260-4484-b5e0-906a26b4fea6\") " Sep 10 14:43:57.441185 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441077 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:43:57.441185 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441092 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:43:57.441342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441219 2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.441342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441238 2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-workload-certs\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.441342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441251 2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-credential-socket\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.441342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441263 2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-data\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.441551 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.441399 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:43:57.442862 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.442823 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 14:43:57.442993 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.442902 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-kube-api-access-mjpcp" (OuterVolumeSpecName: "kube-api-access-mjpcp") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "kube-api-access-mjpcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:43:57.442993 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.442939 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:43:57.442993 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.442955 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-token" (OuterVolumeSpecName: "istio-token") pod "2cd84c3e-4260-4484-b5e0-906a26b4fea6" (UID: "2cd84c3e-4260-4484-b5e0-906a26b4fea6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:43:57.542501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.542468 2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-envoy\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.542501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.542498 2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istiod-ca-cert\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.542501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.542507 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjpcp\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-kube-api-access-mjpcp\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.542736 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.542516 2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-podinfo\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.542736 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.542525 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cd84c3e-4260-4484-b5e0-906a26b4fea6-istio-token\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:43:57.922928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.922897 2576 generic.go:356] "Generic (PLEG): container finished" podID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerID="a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee" exitCode=0 Sep 10 14:43:57.923097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.922943 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" Sep 10 14:43:57.923097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.922965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" event={"ID":"2cd84c3e-4260-4484-b5e0-906a26b4fea6","Type":"ContainerDied","Data":"a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee"} Sep 10 14:43:57.923097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.923005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc" event={"ID":"2cd84c3e-4260-4484-b5e0-906a26b4fea6","Type":"ContainerDied","Data":"92a89fbd49d3ac6f29d4201caa44a1117661706eaa25839c0f0bf24e3f49b0c7"} Sep 10 14:43:57.923097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.923046 2576 scope.go:117] "RemoveContainer" containerID="a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee" Sep 10 14:43:57.931985 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.931967 2576 scope.go:117] "RemoveContainer" containerID="b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09" Sep 10 14:43:57.939740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.939721 2576 scope.go:117] "RemoveContainer" containerID="a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee" Sep 10 14:43:57.940005 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:43:57.939986 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee\": container with ID starting with a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee not found: ID does not exist" containerID="a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee" Sep 10 14:43:57.940050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.940013 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee"} err="failed to get container status \"a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee\": rpc error: code = NotFound desc = could not find container \"a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee\": container with ID starting with a2f55adba6dde22813ec0357bac4f959429f190f25173310ba02ce3b8a9b11ee not found: ID does not exist" Sep 10 14:43:57.940050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.940029 2576 scope.go:117] "RemoveContainer" containerID="b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09" Sep 10 14:43:57.940278 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:43:57.940260 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09\": container with ID starting with b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09 not found: ID does not exist" containerID="b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09" Sep 10 14:43:57.940331 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.940284 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09"} err="failed to get container status \"b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09\": rpc error: code = NotFound desc = could not find container \"b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09\": container with ID starting with b6cd93d14261a10860c26f211fe5426707aff89c3d094376f5360d2c70f29b09 not found: ID does not exist" Sep 10 14:43:57.945008 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.944986 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc"] Sep 10 14:43:57.948142 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:57.948121 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-fc06d4-openshift-default-6bd66998d6-fmxvc"] Sep 10 14:43:58.761866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:43:58.761837 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" path="/var/lib/kubelet/pods/2cd84c3e-4260-4484-b5e0-906a26b4fea6/volumes" Sep 10 14:44:02.338493 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:02.338446 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:05.403837 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:05.403785 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:44:20.915740 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:20.915703 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:44:26.049999 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:26.049963 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:27.131156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:27.131127 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:27.563965 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:27.563887 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:27.867924 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:27.867829 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:28.136663 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:28.136626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:31.189268 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:31.189235 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:31.288302 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:31.288265 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:31.601952 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:31.601861 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:44:32.340443 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.340400 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:32.412691 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.412656 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:32.922555 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922526 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw"] Sep 10 14:44:32.922944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922923 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" containerName="payload-pre-processing" Sep 10 14:44:32.922944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922940 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" containerName="payload-pre-processing" Sep 10 14:44:32.922944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922950 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d36fe7b9-9f57-4215-898b-eb764b353b38" containerName="revoke-keys" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922956 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d36fe7b9-9f57-4215-898b-eb764b353b38" containerName="revoke-keys" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922972 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922979 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922988 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.922993 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.923003 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e387962c-7a02-47de-9312-2efb6e7e57d3" containerName="payload-processing" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.923008 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e387962c-7a02-47de-9312-2efb6e7e57d3" containerName="payload-processing" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.923071 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.923079 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e387962c-7a02-47de-9312-2efb6e7e57d3" containerName="payload-processing" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.923088 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6fc99d-6725-46b4-bc3e-bf95d0cd1541" containerName="payload-pre-processing" Sep 10 14:44:32.923170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.923095 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d36fe7b9-9f57-4215-898b-eb764b353b38" containerName="revoke-keys" Sep 10 14:44:32.926139 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.926120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:32.929029 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.929006 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-42514d\"" Sep 10 14:44:32.929132 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.929044 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-42514d-dockercfg-ppmmx\"" Sep 10 14:44:32.940001 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.939978 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw"] Sep 10 14:44:32.961683 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.961653 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm"] Sep 10 14:44:32.962140 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.962121 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cd84c3e-4260-4484-b5e0-906a26b4fea6" containerName="istio-proxy" Sep 10 14:44:32.965024 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.965005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:32.974602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:32.974579 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm"] Sep 10 14:44:33.061222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.061191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d067463d-2b63-46b2-9bbc-8bc17b278a7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.061222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.061227 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5cgw\" (UniqueName: \"kubernetes.io/projected/578eceea-5259-4fc3-a6eb-408adb68bf69-kube-api-access-j5cgw\") pod \"payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.061430 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.061336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eceea-5259-4fc3-a6eb-408adb68bf69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.061430 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.061385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnn55\" (UniqueName: \"kubernetes.io/projected/d067463d-2b63-46b2-9bbc-8bc17b278a7b-kube-api-access-jnn55\") pod \"payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.162477 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.162432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eceea-5259-4fc3-a6eb-408adb68bf69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.162477 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.162482 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnn55\" (UniqueName: \"kubernetes.io/projected/d067463d-2b63-46b2-9bbc-8bc17b278a7b-kube-api-access-jnn55\") pod \"payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.162723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.162522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d067463d-2b63-46b2-9bbc-8bc17b278a7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.162723 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.162542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5cgw\" (UniqueName: \"kubernetes.io/projected/578eceea-5259-4fc3-a6eb-408adb68bf69-kube-api-access-j5cgw\") pod \"payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.163250 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.163229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eceea-5259-4fc3-a6eb-408adb68bf69-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.163323 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.163265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d067463d-2b63-46b2-9bbc-8bc17b278a7b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.170592 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.170562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5cgw\" (UniqueName: \"kubernetes.io/projected/578eceea-5259-4fc3-a6eb-408adb68bf69-kube-api-access-j5cgw\") pod \"payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.170748 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.170725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnn55\" (UniqueName: \"kubernetes.io/projected/d067463d-2b63-46b2-9bbc-8bc17b278a7b-kube-api-access-jnn55\") pod \"payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.236756 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.236687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:33.275584 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.275556 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:33.374602 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.374569 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:33.432632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.432599 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:33.509011 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.508986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw"] Sep 10 14:44:33.510944 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:44:33.510919 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod578eceea_5259_4fc3_a6eb_408adb68bf69.slice/crio-602f0f488c30ba4f2ce00cc43997519d6835d7a0411b8e44ed5c1d1f921fb09a WatchSource:0}: Error finding container 602f0f488c30ba4f2ce00cc43997519d6835d7a0411b8e44ed5c1d1f921fb09a: Status 404 returned error can't find the container with id 602f0f488c30ba4f2ce00cc43997519d6835d7a0411b8e44ed5c1d1f921fb09a Sep 10 14:44:33.556694 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:33.556501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm"] Sep 10 14:44:33.558638 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:44:33.558613 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd067463d_2b63_46b2_9bbc_8bc17b278a7b.slice/crio-0ec14525af4a05aaaffa1dd95b4525b0df0abeb53e0eb331205e0f5adfba3263 WatchSource:0}: Error finding container 0ec14525af4a05aaaffa1dd95b4525b0df0abeb53e0eb331205e0f5adfba3263: Status 404 returned error can't find the container with id 0ec14525af4a05aaaffa1dd95b4525b0df0abeb53e0eb331205e0f5adfba3263 Sep 10 14:44:34.100618 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.100579 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" event={"ID":"d067463d-2b63-46b2-9bbc-8bc17b278a7b","Type":"ContainerStarted","Data":"428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2"} Sep 10 14:44:34.100841 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.100642 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" event={"ID":"d067463d-2b63-46b2-9bbc-8bc17b278a7b","Type":"ContainerStarted","Data":"0ec14525af4a05aaaffa1dd95b4525b0df0abeb53e0eb331205e0f5adfba3263"} Sep 10 14:44:34.100841 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.100773 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:34.102281 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.102257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" event={"ID":"578eceea-5259-4fc3-a6eb-408adb68bf69","Type":"ContainerStarted","Data":"535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41"} Sep 10 14:44:34.102401 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.102300 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" event={"ID":"578eceea-5259-4fc3-a6eb-408adb68bf69","Type":"ContainerStarted","Data":"602f0f488c30ba4f2ce00cc43997519d6835d7a0411b8e44ed5c1d1f921fb09a"} Sep 10 14:44:34.102401 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.102354 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:34.118187 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.118139 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" podStartSLOduration=2.118123597 podStartE2EDuration="2.118123597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:34.116383146 +0000 UTC m=+1491.970291824" watchObservedRunningTime="2026-09-10 14:44:34.118123597 +0000 UTC m=+1491.972032274" Sep 10 14:44:34.132866 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:34.132819 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" podStartSLOduration=2.132808664 podStartE2EDuration="2.132808664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:44:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:44:34.131120679 +0000 UTC m=+1491.985029358" watchObservedRunningTime="2026-09-10 14:44:34.132808664 +0000 UTC m=+1491.986717334" Sep 10 14:44:41.913405 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:41.913368 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:44:42.646055 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:42.646027 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:44:42.650349 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:42.650327 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:44:45.110415 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:45.110380 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:44:45.110810 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:45.110755 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:44:56.157432 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:56.157393 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:56.839344 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:56.839307 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:57.165128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:57.165091 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:44:57.559719 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:44:57.559630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:00.139569 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.139532 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5"] Sep 10 14:45:00.143174 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.143141 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" Sep 10 14:45:00.144621 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.144594 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s"] Sep 10 14:45:00.145929 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.145909 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:45:00.147762 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.147743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" Sep 10 14:45:00.154517 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.154493 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5"] Sep 10 14:45:00.155347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.155327 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" Sep 10 14:45:00.160282 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.160252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s"] Sep 10 14:45:00.166853 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.166832 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" Sep 10 14:45:00.382631 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.382597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:00.440691 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.440664 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5"] Sep 10 14:45:00.442920 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:45:00.442894 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed2ec44_b546_45fb_959f_ba21549a3532.slice/crio-e551fad4b5d081cacc0611c7ba3d4406eb5cb6fbdc55380ea6cbe876bc2440f3 WatchSource:0}: Error finding container e551fad4b5d081cacc0611c7ba3d4406eb5cb6fbdc55380ea6cbe876bc2440f3: Status 404 returned error can't find the container with id e551fad4b5d081cacc0611c7ba3d4406eb5cb6fbdc55380ea6cbe876bc2440f3 Sep 10 14:45:00.462911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.462881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:00.463820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:00.463774 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s"] Sep 10 14:45:00.475784 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:45:00.475755 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod606f1938_4aa0_486a_a890_9d77f8d3a5a8.slice/crio-d796d79c1500dfbed84eef2825744b4d5e51f7b6343f8d04f38ecb3818f6557b WatchSource:0}: Error finding container d796d79c1500dfbed84eef2825744b4d5e51f7b6343f8d04f38ecb3818f6557b: Status 404 returned error can't find the container with id d796d79c1500dfbed84eef2825744b4d5e51f7b6343f8d04f38ecb3818f6557b Sep 10 14:45:01.251371 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.251335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" event={"ID":"606f1938-4aa0-486a-a890-9d77f8d3a5a8","Type":"ContainerStarted","Data":"e709e56edaf2fac47ca081f38c67dc038be80a25540df5c9befa15200d1c40af"} Sep 10 14:45:01.251824 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.251398 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" event={"ID":"606f1938-4aa0-486a-a890-9d77f8d3a5a8","Type":"ContainerStarted","Data":"d796d79c1500dfbed84eef2825744b4d5e51f7b6343f8d04f38ecb3818f6557b"} Sep 10 14:45:01.252957 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.252924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" event={"ID":"aed2ec44-b546-45fb-959f-ba21549a3532","Type":"ContainerStarted","Data":"1a5d4442f6f572c0f0d825e824a79125520482cd271a844239231a07ba033f16"} Sep 10 14:45:01.253101 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.252983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" event={"ID":"aed2ec44-b546-45fb-959f-ba21549a3532","Type":"ContainerStarted","Data":"e551fad4b5d081cacc0611c7ba3d4406eb5cb6fbdc55380ea6cbe876bc2440f3"} Sep 10 14:45:01.267593 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.267548 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" podStartSLOduration=1.267534645 podStartE2EDuration="1.267534645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:01.26587624 +0000 UTC m=+1519.119784917" watchObservedRunningTime="2026-09-10 14:45:01.267534645 +0000 UTC m=+1519.121443321" Sep 10 14:45:01.281344 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.281300 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" podStartSLOduration=1.281287275 podStartE2EDuration="1.281287275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:01.280097335 +0000 UTC m=+1519.134006024" watchObservedRunningTime="2026-09-10 14:45:01.281287275 +0000 UTC m=+1519.135195951" Sep 10 14:45:01.602913 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.602819 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:01.701905 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:01.701873 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:02.183475 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.183438 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds"] Sep 10 14:45:02.186931 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.186908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.189697 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.189676 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5c43af-dockercfg-lbrkw\"" Sep 10 14:45:02.189823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.189676 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5c43af\"" Sep 10 14:45:02.198162 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.198137 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds"] Sep 10 14:45:02.229444 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.229406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2047fac-4703-46be-9363-175e876be523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.229619 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.229472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w52qk\" (UniqueName: \"kubernetes.io/projected/e2047fac-4703-46be-9363-175e876be523-kube-api-access-w52qk\") pod \"payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.260875 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.260840 2576 generic.go:356] "Generic (PLEG): container finished" podID="606f1938-4aa0-486a-a890-9d77f8d3a5a8" containerID="e709e56edaf2fac47ca081f38c67dc038be80a25540df5c9befa15200d1c40af" exitCode=0 Sep 10 14:45:02.261318 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.260920 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" event={"ID":"606f1938-4aa0-486a-a890-9d77f8d3a5a8","Type":"ContainerDied","Data":"e709e56edaf2fac47ca081f38c67dc038be80a25540df5c9befa15200d1c40af"} Sep 10 14:45:02.262950 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.262922 2576 generic.go:356] "Generic (PLEG): container finished" podID="aed2ec44-b546-45fb-959f-ba21549a3532" containerID="1a5d4442f6f572c0f0d825e824a79125520482cd271a844239231a07ba033f16" exitCode=0 Sep 10 14:45:02.263089 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.262971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" event={"ID":"aed2ec44-b546-45fb-959f-ba21549a3532","Type":"ContainerDied","Data":"1a5d4442f6f572c0f0d825e824a79125520482cd271a844239231a07ba033f16"} Sep 10 14:45:02.283746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.283712 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:02.330088 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.330052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2047fac-4703-46be-9363-175e876be523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.330284 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.330135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w52qk\" (UniqueName: \"kubernetes.io/projected/e2047fac-4703-46be-9363-175e876be523-kube-api-access-w52qk\") pod \"payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.330758 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.330737 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2047fac-4703-46be-9363-175e876be523-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.339513 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.339472 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w52qk\" (UniqueName: \"kubernetes.io/projected/e2047fac-4703-46be-9363-175e876be523-kube-api-access-w52qk\") pod \"payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.377580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.377542 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:02.499424 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.499333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:02.623260 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.623226 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:02.776415 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:02.776394 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds"] Sep 10 14:45:02.778817 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:45:02.778774 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2047fac_4703_46be_9363_175e876be523.slice/crio-1b8841d56d5065292ef3ac8c120ecc6febd51106850737ae40096004f77f991d WatchSource:0}: Error finding container 1b8841d56d5065292ef3ac8c120ecc6febd51106850737ae40096004f77f991d: Status 404 returned error can't find the container with id 1b8841d56d5065292ef3ac8c120ecc6febd51106850737ae40096004f77f991d Sep 10 14:45:03.270871 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:03.270831 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" event={"ID":"e2047fac-4703-46be-9363-175e876be523","Type":"ContainerStarted","Data":"e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3"} Sep 10 14:45:03.271365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:03.270896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" event={"ID":"e2047fac-4703-46be-9363-175e876be523","Type":"ContainerStarted","Data":"1b8841d56d5065292ef3ac8c120ecc6febd51106850737ae40096004f77f991d"} Sep 10 14:45:03.295597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:03.292068 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" podStartSLOduration=1.292042243 podStartE2EDuration="1.292042243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:45:03.28737539 +0000 UTC m=+1521.141284071" watchObservedRunningTime="2026-09-10 14:45:03.292042243 +0000 UTC m=+1521.145950923" Sep 10 14:45:03.414347 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:03.414319 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" Sep 10 14:45:03.445525 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:03.445498 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" Sep 10 14:45:04.279185 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.279153 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" Sep 10 14:45:04.279662 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.279152 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s" event={"ID":"606f1938-4aa0-486a-a890-9d77f8d3a5a8","Type":"ContainerDied","Data":"d796d79c1500dfbed84eef2825744b4d5e51f7b6343f8d04f38ecb3818f6557b"} Sep 10 14:45:04.279662 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.279280 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d796d79c1500dfbed84eef2825744b4d5e51f7b6343f8d04f38ecb3818f6557b" Sep 10 14:45:04.280601 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.280581 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" Sep 10 14:45:04.280722 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.280620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5" event={"ID":"aed2ec44-b546-45fb-959f-ba21549a3532","Type":"ContainerDied","Data":"e551fad4b5d081cacc0611c7ba3d4406eb5cb6fbdc55380ea6cbe876bc2440f3"} Sep 10 14:45:04.280722 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.280656 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e551fad4b5d081cacc0611c7ba3d4406eb5cb6fbdc55380ea6cbe876bc2440f3" Sep 10 14:45:04.280916 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:04.280899 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:08.212931 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:08.212848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:45:15.288223 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:15.288189 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:45:25.355121 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:25.355079 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:30.542357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:30.542317 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds_e2047fac-4703-46be-9363-175e876be523/payload-processing/0.log" Sep 10 14:45:36.235412 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:36.235371 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:45:40.215029 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:40.214986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:45:42.837603 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:42.837571 2576 scope.go:117] "RemoveContainer" containerID="9845d2a1710de75fb9eca43b8d6ceb27be28a4e51f3eb15ca31afe82fe81bfa4" Sep 10 14:45:54.510569 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:45:54.510533 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:46:00.955093 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:00.955060 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm_d067463d-2b63-46b2-9bbc-8bc17b278a7b/payload-processing/0.log" Sep 10 14:46:01.187747 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:01.187712 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds_e2047fac-4703-46be-9363-175e876be523/payload-processing/0.log" Sep 10 14:46:01.459108 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:01.459071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:01.761496 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:01.761408 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:01.917963 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:01.917920 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:04.759426 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:04.759349 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:04.853412 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:04.853375 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:06.160276 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.160239 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:06.305203 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.305157 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:06.398947 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.398910 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:06.814123 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814092 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9"] Sep 10 14:46:06.814435 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814423 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="606f1938-4aa0-486a-a890-9d77f8d3a5a8" containerName="cleanup" Sep 10 14:46:06.814480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814437 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="606f1938-4aa0-486a-a890-9d77f8d3a5a8" containerName="cleanup" Sep 10 14:46:06.814480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814452 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed2ec44-b546-45fb-959f-ba21549a3532" containerName="cleanup" Sep 10 14:46:06.814480 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814458 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed2ec44-b546-45fb-959f-ba21549a3532" containerName="cleanup" Sep 10 14:46:06.814578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814527 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="606f1938-4aa0-486a-a890-9d77f8d3a5a8" containerName="cleanup" Sep 10 14:46:06.814578 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.814538 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed2ec44-b546-45fb-959f-ba21549a3532" containerName="cleanup" Sep 10 14:46:06.817641 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.817625 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:06.820265 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.820242 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-268239\"" Sep 10 14:46:06.820385 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.820300 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-268239-dockercfg-sp29p\"" Sep 10 14:46:06.827048 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.827026 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9"] Sep 10 14:46:06.909166 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.909133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mcl\" (UniqueName: \"kubernetes.io/projected/36f9db43-f2a5-4eab-9fb5-0361b8276dae-kube-api-access-v7mcl\") pod \"payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:06.909332 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:06.909198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f9db43-f2a5-4eab-9fb5-0361b8276dae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.010143 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.010111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mcl\" (UniqueName: \"kubernetes.io/projected/36f9db43-f2a5-4eab-9fb5-0361b8276dae-kube-api-access-v7mcl\") pod \"payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.010300 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.010165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f9db43-f2a5-4eab-9fb5-0361b8276dae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.010863 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.010846 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f9db43-f2a5-4eab-9fb5-0361b8276dae-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.018175 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.018153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mcl\" (UniqueName: \"kubernetes.io/projected/36f9db43-f2a5-4eab-9fb5-0361b8276dae-kube-api-access-v7mcl\") pod \"payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.129755 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.129728 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.403288 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.403169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9"] Sep 10 14:46:07.406059 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:46:07.406030 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f9db43_f2a5_4eab_9fb5_0361b8276dae.slice/crio-8c6b5db6e854ea326b43981291b4c951245d3fe3a3d04dbb2c33a88f7242bcf6 WatchSource:0}: Error finding container 8c6b5db6e854ea326b43981291b4c951245d3fe3a3d04dbb2c33a88f7242bcf6: Status 404 returned error can't find the container with id 8c6b5db6e854ea326b43981291b4c951245d3fe3a3d04dbb2c33a88f7242bcf6 Sep 10 14:46:07.531134 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.531084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:07.607418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.607376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" event={"ID":"36f9db43-f2a5-4eab-9fb5-0361b8276dae","Type":"ContainerStarted","Data":"36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9"} Sep 10 14:46:07.607597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.607438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" event={"ID":"36f9db43-f2a5-4eab-9fb5-0361b8276dae","Type":"ContainerStarted","Data":"8c6b5db6e854ea326b43981291b4c951245d3fe3a3d04dbb2c33a88f7242bcf6"} Sep 10 14:46:07.607597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.607467 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:07.625718 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.625672 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" podStartSLOduration=1.625657408 podStartE2EDuration="1.625657408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:07.625134064 +0000 UTC m=+1585.479042733" watchObservedRunningTime="2026-09-10 14:46:07.625657408 +0000 UTC m=+1585.479566154" Sep 10 14:46:07.629368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:07.629333 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:13.911620 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:13.911580 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:46:18.616526 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:18.616493 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:29.736544 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:29.736507 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:30.401879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.401841 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8"] Sep 10 14:46:30.405228 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.405211 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:30.408254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.408233 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:46:30.408414 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.408233 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:46:30.413220 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.413199 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8"] Sep 10 14:46:30.417980 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.417957 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08cbc13-3f83-4841-b22b-2075f6f8857e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8\" (UID: \"e08cbc13-3f83-4841-b22b-2075f6f8857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:30.518390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.518355 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08cbc13-3f83-4841-b22b-2075f6f8857e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8\" (UID: \"e08cbc13-3f83-4841-b22b-2075f6f8857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:30.519011 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.518992 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08cbc13-3f83-4841-b22b-2075f6f8857e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8\" (UID: \"e08cbc13-3f83-4841-b22b-2075f6f8857e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:30.716883 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.716773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:30.988547 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:30.988521 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8"] Sep 10 14:46:30.990266 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:46:30.990242 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08cbc13_3f83_4841_b22b_2075f6f8857e.slice/crio-cb57a4904fadd91f70dbbfb5aa3bc7be7032bba140f9ab76f82663edab013ae3 WatchSource:0}: Error finding container cb57a4904fadd91f70dbbfb5aa3bc7be7032bba140f9ab76f82663edab013ae3: Status 404 returned error can't find the container with id cb57a4904fadd91f70dbbfb5aa3bc7be7032bba140f9ab76f82663edab013ae3 Sep 10 14:46:31.735514 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:31.735487 2576 generic.go:356] "Generic (PLEG): container finished" podID="e08cbc13-3f83-4841-b22b-2075f6f8857e" containerID="14c4190b771a189b3e12f43d0123c0934ce5bd011ebcdbc60dd006e7e95faf3d" exitCode=0 Sep 10 14:46:31.735678 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:31.735569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" event={"ID":"e08cbc13-3f83-4841-b22b-2075f6f8857e","Type":"ContainerDied","Data":"14c4190b771a189b3e12f43d0123c0934ce5bd011ebcdbc60dd006e7e95faf3d"} Sep 10 14:46:31.735678 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:31.735628 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" event={"ID":"e08cbc13-3f83-4841-b22b-2075f6f8857e","Type":"ContainerStarted","Data":"cb57a4904fadd91f70dbbfb5aa3bc7be7032bba140f9ab76f82663edab013ae3"} Sep 10 14:46:32.871823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:32.871784 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:32.939557 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:32.939530 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08cbc13-3f83-4841-b22b-2075f6f8857e-maas-api-service-ca\") pod \"e08cbc13-3f83-4841-b22b-2075f6f8857e\" (UID: \"e08cbc13-3f83-4841-b22b-2075f6f8857e\") " Sep 10 14:46:32.939849 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:32.939830 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08cbc13-3f83-4841-b22b-2075f6f8857e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e08cbc13-3f83-4841-b22b-2075f6f8857e" (UID: "e08cbc13-3f83-4841-b22b-2075f6f8857e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:46:33.041078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:33.041000 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08cbc13-3f83-4841-b22b-2075f6f8857e-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:46:33.747194 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:33.747156 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" event={"ID":"e08cbc13-3f83-4841-b22b-2075f6f8857e","Type":"ContainerDied","Data":"cb57a4904fadd91f70dbbfb5aa3bc7be7032bba140f9ab76f82663edab013ae3"} Sep 10 14:46:33.747365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:33.747220 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb57a4904fadd91f70dbbfb5aa3bc7be7032bba140f9ab76f82663edab013ae3" Sep 10 14:46:33.747365 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:33.747168 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8" Sep 10 14:46:36.382118 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.382038 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:36.411986 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.411952 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9"] Sep 10 14:46:36.412190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.412165 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" podUID="36f9db43-f2a5-4eab-9fb5-0361b8276dae" containerName="payload-processing" containerID="cri-o://36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9" gracePeriod=30 Sep 10 14:46:36.462359 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.462316 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:36.561209 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.561167 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:36.573397 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.573366 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8"] Sep 10 14:46:36.577305 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.577278 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-268239-6j4n8"] Sep 10 14:46:36.635607 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.635499 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:36.661221 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.661200 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:36.762847 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.762782 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e08cbc13-3f83-4841-b22b-2075f6f8857e" path="/var/lib/kubelet/pods/e08cbc13-3f83-4841-b22b-2075f6f8857e/volumes" Sep 10 14:46:36.763572 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.763545 2576 generic.go:356] "Generic (PLEG): container finished" podID="36f9db43-f2a5-4eab-9fb5-0361b8276dae" containerID="36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9" exitCode=0 Sep 10 14:46:36.763726 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.763659 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" Sep 10 14:46:36.765048 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.765027 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" event={"ID":"36f9db43-f2a5-4eab-9fb5-0361b8276dae","Type":"ContainerDied","Data":"36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9"} Sep 10 14:46:36.765170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.765070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9" event={"ID":"36f9db43-f2a5-4eab-9fb5-0361b8276dae","Type":"ContainerDied","Data":"8c6b5db6e854ea326b43981291b4c951245d3fe3a3d04dbb2c33a88f7242bcf6"} Sep 10 14:46:36.765170 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.765092 2576 scope.go:117] "RemoveContainer" containerID="36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9" Sep 10 14:46:36.773294 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.773278 2576 scope.go:117] "RemoveContainer" containerID="36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9" Sep 10 14:46:36.773580 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:46:36.773555 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9\": container with ID starting with 36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9 not found: ID does not exist" containerID="36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9" Sep 10 14:46:36.773636 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.773586 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9"} err="failed to get container status \"36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9\": rpc error: code = NotFound desc = could not find container \"36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9\": container with ID starting with 36a314479c14c029b4bb594b5a232fd036d597309ebd7a99a65b3d0aa30ffaa9 not found: ID does not exist" Sep 10 14:46:36.774777 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.774760 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7mcl\" (UniqueName: \"kubernetes.io/projected/36f9db43-f2a5-4eab-9fb5-0361b8276dae-kube-api-access-v7mcl\") pod \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " Sep 10 14:46:36.774915 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.774899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f9db43-f2a5-4eab-9fb5-0361b8276dae-plugins-config-volume\") pod \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\" (UID: \"36f9db43-f2a5-4eab-9fb5-0361b8276dae\") " Sep 10 14:46:36.775254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.775233 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f9db43-f2a5-4eab-9fb5-0361b8276dae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36f9db43-f2a5-4eab-9fb5-0361b8276dae" (UID: "36f9db43-f2a5-4eab-9fb5-0361b8276dae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:46:36.776585 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.776563 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f9db43-f2a5-4eab-9fb5-0361b8276dae-kube-api-access-v7mcl" (OuterVolumeSpecName: "kube-api-access-v7mcl") pod "36f9db43-f2a5-4eab-9fb5-0361b8276dae" (UID: "36f9db43-f2a5-4eab-9fb5-0361b8276dae"). InnerVolumeSpecName "kube-api-access-v7mcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:46:36.876560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.876532 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7mcl\" (UniqueName: \"kubernetes.io/projected/36f9db43-f2a5-4eab-9fb5-0361b8276dae-kube-api-access-v7mcl\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:46:36.876560 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.876556 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36f9db43-f2a5-4eab-9fb5-0361b8276dae-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:46:36.945770 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:36.945683 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:37.086043 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:37.086015 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9"] Sep 10 14:46:37.090278 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:37.090253 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-268239-65fd74c747-7r9k9"] Sep 10 14:46:38.761626 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:38.761590 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36f9db43-f2a5-4eab-9fb5-0361b8276dae" path="/var/lib/kubelet/pods/36f9db43-f2a5-4eab-9fb5-0361b8276dae/volumes" Sep 10 14:46:41.009177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:41.009139 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:46:42.921212 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:42.921178 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:43.046313 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:43.046274 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:43.187546 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:43.187509 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:46.716854 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.716813 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5"] Sep 10 14:46:46.717246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.717212 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08cbc13-3f83-4841-b22b-2075f6f8857e" containerName="revoke-keys" Sep 10 14:46:46.717246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.717224 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08cbc13-3f83-4841-b22b-2075f6f8857e" containerName="revoke-keys" Sep 10 14:46:46.717246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.717235 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36f9db43-f2a5-4eab-9fb5-0361b8276dae" containerName="payload-processing" Sep 10 14:46:46.717246 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.717241 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="36f9db43-f2a5-4eab-9fb5-0361b8276dae" containerName="payload-processing" Sep 10 14:46:46.717402 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.717296 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08cbc13-3f83-4841-b22b-2075f6f8857e" containerName="revoke-keys" Sep 10 14:46:46.717402 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.717308 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="36f9db43-f2a5-4eab-9fb5-0361b8276dae" containerName="payload-processing" Sep 10 14:46:46.720019 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.720002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:46.722711 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.722683 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:46:46.722856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.722683 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:46:46.727597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.727574 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5"] Sep 10 14:46:46.761141 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.761103 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52748089-0120-4156-9206-cc599026f35e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5\" (UID: \"52748089-0120-4156-9206-cc599026f35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:46.862548 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.862505 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52748089-0120-4156-9206-cc599026f35e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5\" (UID: \"52748089-0120-4156-9206-cc599026f35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:46.863201 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:46.863173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52748089-0120-4156-9206-cc599026f35e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5\" (UID: \"52748089-0120-4156-9206-cc599026f35e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:47.021900 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:47.021787 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:46:47.031744 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:47.031717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:47.306452 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:47.306423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5"] Sep 10 14:46:47.309234 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:46:47.309198 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52748089_0120_4156_9206_cc599026f35e.slice/crio-734a9f89f8fd3f4c4fb930159a642c60af211e77ea5009602a274a6979d44921 WatchSource:0}: Error finding container 734a9f89f8fd3f4c4fb930159a642c60af211e77ea5009602a274a6979d44921: Status 404 returned error can't find the container with id 734a9f89f8fd3f4c4fb930159a642c60af211e77ea5009602a274a6979d44921 Sep 10 14:46:47.820531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:47.820499 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" event={"ID":"52748089-0120-4156-9206-cc599026f35e","Type":"ContainerStarted","Data":"1fabff5d8f88b849c5fafc0d2cf64cefc09983cc987cfa75c63753c96af0a024"} Sep 10 14:46:47.820928 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:47.820550 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" event={"ID":"52748089-0120-4156-9206-cc599026f35e","Type":"ContainerStarted","Data":"734a9f89f8fd3f4c4fb930159a642c60af211e77ea5009602a274a6979d44921"} Sep 10 14:46:47.835763 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:47.835721 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" podStartSLOduration=1.835708868 podStartE2EDuration="1.835708868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:46:47.834301095 +0000 UTC m=+1625.688209771" watchObservedRunningTime="2026-09-10 14:46:47.835708868 +0000 UTC m=+1625.689617544" Sep 10 14:46:48.827097 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:48.827065 2576 generic.go:356] "Generic (PLEG): container finished" podID="52748089-0120-4156-9206-cc599026f35e" containerID="1fabff5d8f88b849c5fafc0d2cf64cefc09983cc987cfa75c63753c96af0a024" exitCode=0 Sep 10 14:46:48.827474 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:48.827150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" event={"ID":"52748089-0120-4156-9206-cc599026f35e","Type":"ContainerDied","Data":"1fabff5d8f88b849c5fafc0d2cf64cefc09983cc987cfa75c63753c96af0a024"} Sep 10 14:46:49.968634 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:49.968610 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:50.094916 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:50.094836 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52748089-0120-4156-9206-cc599026f35e-maas-api-service-ca\") pod \"52748089-0120-4156-9206-cc599026f35e\" (UID: \"52748089-0120-4156-9206-cc599026f35e\") " Sep 10 14:46:50.095166 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:50.095144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52748089-0120-4156-9206-cc599026f35e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52748089-0120-4156-9206-cc599026f35e" (UID: "52748089-0120-4156-9206-cc599026f35e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:46:50.195774 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:50.195741 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52748089-0120-4156-9206-cc599026f35e-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:46:50.837887 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:50.837851 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" Sep 10 14:46:50.838037 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:50.837885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5" event={"ID":"52748089-0120-4156-9206-cc599026f35e","Type":"ContainerDied","Data":"734a9f89f8fd3f4c4fb930159a642c60af211e77ea5009602a274a6979d44921"} Sep 10 14:46:50.838037 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:50.837932 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734a9f89f8fd3f4c4fb930159a642c60af211e77ea5009602a274a6979d44921" Sep 10 14:46:59.912019 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:46:59.911981 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:47:02.249590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.249557 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:02.270370 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.270335 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm"] Sep 10 14:47:02.270651 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.270620 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" podUID="d067463d-2b63-46b2-9bbc-8bc17b278a7b" containerName="payload-processing" containerID="cri-o://428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2" gracePeriod=30 Sep 10 14:47:02.278889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.278859 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw"] Sep 10 14:47:02.279214 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.279184 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" podUID="578eceea-5259-4fc3-a6eb-408adb68bf69" containerName="payload-pre-processing" containerID="cri-o://535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41" gracePeriod=30 Sep 10 14:47:02.314891 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.314856 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:02.428999 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.428935 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5"] Sep 10 14:47:02.431402 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.431374 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-42514d-rbpn5"] Sep 10 14:47:02.431550 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.431366 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:02.490955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.490928 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:02.534977 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.534953 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:47:02.561582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.561555 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:47:02.712368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.712326 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5cgw\" (UniqueName: \"kubernetes.io/projected/578eceea-5259-4fc3-a6eb-408adb68bf69-kube-api-access-j5cgw\") pod \"578eceea-5259-4fc3-a6eb-408adb68bf69\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " Sep 10 14:47:02.712368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.712376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnn55\" (UniqueName: \"kubernetes.io/projected/d067463d-2b63-46b2-9bbc-8bc17b278a7b-kube-api-access-jnn55\") pod \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " Sep 10 14:47:02.712582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.712423 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eceea-5259-4fc3-a6eb-408adb68bf69-plugins-config-volume\") pod \"578eceea-5259-4fc3-a6eb-408adb68bf69\" (UID: \"578eceea-5259-4fc3-a6eb-408adb68bf69\") " Sep 10 14:47:02.712582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.712442 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d067463d-2b63-46b2-9bbc-8bc17b278a7b-plugins-config-volume\") pod \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\" (UID: \"d067463d-2b63-46b2-9bbc-8bc17b278a7b\") " Sep 10 14:47:02.712856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.712815 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/578eceea-5259-4fc3-a6eb-408adb68bf69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "578eceea-5259-4fc3-a6eb-408adb68bf69" (UID: "578eceea-5259-4fc3-a6eb-408adb68bf69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:47:02.712856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.712853 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d067463d-2b63-46b2-9bbc-8bc17b278a7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d067463d-2b63-46b2-9bbc-8bc17b278a7b" (UID: "d067463d-2b63-46b2-9bbc-8bc17b278a7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:47:02.714450 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.714430 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d067463d-2b63-46b2-9bbc-8bc17b278a7b-kube-api-access-jnn55" (OuterVolumeSpecName: "kube-api-access-jnn55") pod "d067463d-2b63-46b2-9bbc-8bc17b278a7b" (UID: "d067463d-2b63-46b2-9bbc-8bc17b278a7b"). InnerVolumeSpecName "kube-api-access-jnn55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:47:02.714504 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.714491 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/578eceea-5259-4fc3-a6eb-408adb68bf69-kube-api-access-j5cgw" (OuterVolumeSpecName: "kube-api-access-j5cgw") pod "578eceea-5259-4fc3-a6eb-408adb68bf69" (UID: "578eceea-5259-4fc3-a6eb-408adb68bf69"). InnerVolumeSpecName "kube-api-access-j5cgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:47:02.762051 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.761979 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52748089-0120-4156-9206-cc599026f35e" path="/var/lib/kubelet/pods/52748089-0120-4156-9206-cc599026f35e/volumes" Sep 10 14:47:02.799489 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.799461 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:02.813286 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.813264 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5cgw\" (UniqueName: \"kubernetes.io/projected/578eceea-5259-4fc3-a6eb-408adb68bf69-kube-api-access-j5cgw\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:47:02.813286 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.813285 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnn55\" (UniqueName: \"kubernetes.io/projected/d067463d-2b63-46b2-9bbc-8bc17b278a7b-kube-api-access-jnn55\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:47:02.813418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.813294 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/578eceea-5259-4fc3-a6eb-408adb68bf69-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:47:02.813418 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.813303 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d067463d-2b63-46b2-9bbc-8bc17b278a7b-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:47:02.902820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.902307 2576 generic.go:356] "Generic (PLEG): container finished" podID="d067463d-2b63-46b2-9bbc-8bc17b278a7b" containerID="428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2" exitCode=0 Sep 10 14:47:02.902820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.902488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" event={"ID":"d067463d-2b63-46b2-9bbc-8bc17b278a7b","Type":"ContainerDied","Data":"428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2"} Sep 10 14:47:02.902820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.902533 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" Sep 10 14:47:02.902820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.902598 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm" event={"ID":"d067463d-2b63-46b2-9bbc-8bc17b278a7b","Type":"ContainerDied","Data":"0ec14525af4a05aaaffa1dd95b4525b0df0abeb53e0eb331205e0f5adfba3263"} Sep 10 14:47:02.902820 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.902645 2576 scope.go:117] "RemoveContainer" containerID="428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2" Sep 10 14:47:02.909454 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.909379 2576 generic.go:356] "Generic (PLEG): container finished" podID="578eceea-5259-4fc3-a6eb-408adb68bf69" containerID="535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41" exitCode=0 Sep 10 14:47:02.909529 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.909512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" event={"ID":"578eceea-5259-4fc3-a6eb-408adb68bf69","Type":"ContainerDied","Data":"535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41"} Sep 10 14:47:02.909589 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.909559 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" event={"ID":"578eceea-5259-4fc3-a6eb-408adb68bf69","Type":"ContainerDied","Data":"602f0f488c30ba4f2ce00cc43997519d6835d7a0411b8e44ed5c1d1f921fb09a"} Sep 10 14:47:02.909901 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.909879 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw" Sep 10 14:47:02.917190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.917171 2576 scope.go:117] "RemoveContainer" containerID="428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2" Sep 10 14:47:02.917450 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:47:02.917429 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2\": container with ID starting with 428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2 not found: ID does not exist" containerID="428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2" Sep 10 14:47:02.917528 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.917462 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2"} err="failed to get container status \"428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2\": rpc error: code = NotFound desc = could not find container \"428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2\": container with ID starting with 428abc1b88bdebae48128a3abd0354a187b28d4bf572f7ff0db9aff4033ccaa2 not found: ID does not exist" Sep 10 14:47:02.917528 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.917486 2576 scope.go:117] "RemoveContainer" containerID="535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41" Sep 10 14:47:02.925253 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.925236 2576 scope.go:117] "RemoveContainer" containerID="535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41" Sep 10 14:47:02.925554 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:47:02.925531 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41\": container with ID starting with 535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41 not found: ID does not exist" containerID="535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41" Sep 10 14:47:02.925632 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.925562 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41"} err="failed to get container status \"535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41\": rpc error: code = NotFound desc = could not find container \"535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41\": container with ID starting with 535e579cb90f871cf565c3c4c7bd742b7bfbae74310be00bea9f9a4de3322b41 not found: ID does not exist" Sep 10 14:47:02.926832 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.926809 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm"] Sep 10 14:47:02.928582 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.928560 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-42514d-74ddb96797-bh6dm"] Sep 10 14:47:02.938914 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.938894 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw"] Sep 10 14:47:02.942944 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:02.942924 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-42514d-56dc6c9bf8-q9bqw"] Sep 10 14:47:04.760868 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:04.760834 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="578eceea-5259-4fc3-a6eb-408adb68bf69" path="/var/lib/kubelet/pods/578eceea-5259-4fc3-a6eb-408adb68bf69/volumes" Sep 10 14:47:04.761325 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:04.761193 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d067463d-2b63-46b2-9bbc-8bc17b278a7b" path="/var/lib/kubelet/pods/d067463d-2b63-46b2-9bbc-8bc17b278a7b/volumes" Sep 10 14:47:07.826872 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:07.826832 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:13.720614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:13.720578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:47:22.863540 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:22.863471 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:22.964406 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:22.964348 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:25.334436 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:25.334396 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:44.051576 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:44.051521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:44.738454 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:44.738424 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:50.324110 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.324071 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:50.388638 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.388604 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:50.630739 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.630707 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:50.972663 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.972572 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx"] Sep 10 14:47:50.973100 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973080 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52748089-0120-4156-9206-cc599026f35e" containerName="revoke-keys" Sep 10 14:47:50.973224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973101 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="52748089-0120-4156-9206-cc599026f35e" containerName="revoke-keys" Sep 10 14:47:50.973224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973124 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="578eceea-5259-4fc3-a6eb-408adb68bf69" containerName="payload-pre-processing" Sep 10 14:47:50.973224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973133 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="578eceea-5259-4fc3-a6eb-408adb68bf69" containerName="payload-pre-processing" Sep 10 14:47:50.973224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973170 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d067463d-2b63-46b2-9bbc-8bc17b278a7b" containerName="payload-processing" Sep 10 14:47:50.973224 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973179 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d067463d-2b63-46b2-9bbc-8bc17b278a7b" containerName="payload-processing" Sep 10 14:47:50.973484 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973307 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d067463d-2b63-46b2-9bbc-8bc17b278a7b" containerName="payload-processing" Sep 10 14:47:50.973484 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973323 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="52748089-0120-4156-9206-cc599026f35e" containerName="revoke-keys" Sep 10 14:47:50.973484 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.973333 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="578eceea-5259-4fc3-a6eb-408adb68bf69" containerName="payload-pre-processing" Sep 10 14:47:50.976469 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.976449 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:50.981242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.981226 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:47:50.981364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.981229 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:47:50.990448 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:50.990424 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx"] Sep 10 14:47:51.138596 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:51.138556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36fd75d-4b3c-4671-a810-89d846acc932-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx\" (UID: \"a36fd75d-4b3c-4671-a810-89d846acc932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:51.218721 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:51.218679 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:51.240004 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:51.239936 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36fd75d-4b3c-4671-a810-89d846acc932-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx\" (UID: \"a36fd75d-4b3c-4671-a810-89d846acc932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:51.240517 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:51.240497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36fd75d-4b3c-4671-a810-89d846acc932-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx\" (UID: \"a36fd75d-4b3c-4671-a810-89d846acc932\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:51.286929 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:51.286900 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:51.560523 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:51.560495 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx"] Sep 10 14:47:51.562766 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:47:51.562738 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36fd75d_4b3c_4671_a810_89d846acc932.slice/crio-5be78610fb08c4b6cb26e9f677d893ad255c082f693bfef3537ff03344d2296c WatchSource:0}: Error finding container 5be78610fb08c4b6cb26e9f677d893ad255c082f693bfef3537ff03344d2296c: Status 404 returned error can't find the container with id 5be78610fb08c4b6cb26e9f677d893ad255c082f693bfef3537ff03344d2296c Sep 10 14:47:52.170507 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:52.170471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" event={"ID":"a36fd75d-4b3c-4671-a810-89d846acc932","Type":"ContainerStarted","Data":"9cd9cb8a03b10a0467aaeb4cf0ab057bfb9a8dba6fdcd313a4a189868d36393d"} Sep 10 14:47:52.170684 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:52.170531 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" event={"ID":"a36fd75d-4b3c-4671-a810-89d846acc932","Type":"ContainerStarted","Data":"5be78610fb08c4b6cb26e9f677d893ad255c082f693bfef3537ff03344d2296c"} Sep 10 14:47:52.188969 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:52.188922 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" podStartSLOduration=2.188908062 podStartE2EDuration="2.188908062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:47:52.187850639 +0000 UTC m=+1690.041759319" watchObservedRunningTime="2026-09-10 14:47:52.188908062 +0000 UTC m=+1690.042816737" Sep 10 14:47:53.177055 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:53.177021 2576 generic.go:356] "Generic (PLEG): container finished" podID="a36fd75d-4b3c-4671-a810-89d846acc932" containerID="9cd9cb8a03b10a0467aaeb4cf0ab057bfb9a8dba6fdcd313a4a189868d36393d" exitCode=0 Sep 10 14:47:53.177410 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:53.177101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" event={"ID":"a36fd75d-4b3c-4671-a810-89d846acc932","Type":"ContainerDied","Data":"9cd9cb8a03b10a0467aaeb4cf0ab057bfb9a8dba6fdcd313a4a189868d36393d"} Sep 10 14:47:53.443730 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:53.443642 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:47:54.305344 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:54.305319 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:54.467648 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:54.467560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36fd75d-4b3c-4671-a810-89d846acc932-maas-api-service-ca\") pod \"a36fd75d-4b3c-4671-a810-89d846acc932\" (UID: \"a36fd75d-4b3c-4671-a810-89d846acc932\") " Sep 10 14:47:54.467990 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:54.467959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36fd75d-4b3c-4671-a810-89d846acc932-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a36fd75d-4b3c-4671-a810-89d846acc932" (UID: "a36fd75d-4b3c-4671-a810-89d846acc932"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:47:54.568559 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:54.568518 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a36fd75d-4b3c-4671-a810-89d846acc932-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:47:55.188501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:55.188462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" event={"ID":"a36fd75d-4b3c-4671-a810-89d846acc932","Type":"ContainerDied","Data":"5be78610fb08c4b6cb26e9f677d893ad255c082f693bfef3537ff03344d2296c"} Sep 10 14:47:55.188501 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:55.188486 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx" Sep 10 14:47:55.188713 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:47:55.188519 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5be78610fb08c4b6cb26e9f677d893ad255c082f693bfef3537ff03344d2296c" Sep 10 14:48:01.228064 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:01.228006 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:01.745559 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:01.745520 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:01.808543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:01.808510 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds"] Sep 10 14:48:01.808805 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:01.808764 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" podUID="e2047fac-4703-46be-9363-175e876be523" containerName="payload-processing" containerID="cri-o://e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3" gracePeriod=30 Sep 10 14:48:01.876078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:01.876046 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:01.992081 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:01.992045 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:02.075392 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.075369 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:48:02.134966 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.134939 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2047fac-4703-46be-9363-175e876be523-plugins-config-volume\") pod \"e2047fac-4703-46be-9363-175e876be523\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " Sep 10 14:48:02.135106 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.134991 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w52qk\" (UniqueName: \"kubernetes.io/projected/e2047fac-4703-46be-9363-175e876be523-kube-api-access-w52qk\") pod \"e2047fac-4703-46be-9363-175e876be523\" (UID: \"e2047fac-4703-46be-9363-175e876be523\") " Sep 10 14:48:02.135306 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.135283 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2047fac-4703-46be-9363-175e876be523-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2047fac-4703-46be-9363-175e876be523" (UID: "e2047fac-4703-46be-9363-175e876be523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:48:02.137132 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.137101 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2047fac-4703-46be-9363-175e876be523-kube-api-access-w52qk" (OuterVolumeSpecName: "kube-api-access-w52qk") pod "e2047fac-4703-46be-9363-175e876be523" (UID: "e2047fac-4703-46be-9363-175e876be523"). InnerVolumeSpecName "kube-api-access-w52qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:48:02.156242 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.156211 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:02.176764 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.176734 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx"] Sep 10 14:48:02.178930 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.178906 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5c43af-zp7sx"] Sep 10 14:48:02.225865 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.225808 2576 generic.go:356] "Generic (PLEG): container finished" podID="e2047fac-4703-46be-9363-175e876be523" containerID="e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3" exitCode=0 Sep 10 14:48:02.225865 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.225856 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" Sep 10 14:48:02.226068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.225884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" event={"ID":"e2047fac-4703-46be-9363-175e876be523","Type":"ContainerDied","Data":"e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3"} Sep 10 14:48:02.226068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.225923 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds" event={"ID":"e2047fac-4703-46be-9363-175e876be523","Type":"ContainerDied","Data":"1b8841d56d5065292ef3ac8c120ecc6febd51106850737ae40096004f77f991d"} Sep 10 14:48:02.226068 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.225946 2576 scope.go:117] "RemoveContainer" containerID="e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3" Sep 10 14:48:02.234715 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.234696 2576 scope.go:117] "RemoveContainer" containerID="e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3" Sep 10 14:48:02.235052 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:48:02.234968 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3\": container with ID starting with e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3 not found: ID does not exist" containerID="e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3" Sep 10 14:48:02.235052 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.234991 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3"} err="failed to get container status \"e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3\": rpc error: code = NotFound desc = could not find container \"e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3\": container with ID starting with e964a9d1d3ba7b4bcc8a7c9de2562fcea4419234f748331de240b81c4332f2d3 not found: ID does not exist" Sep 10 14:48:02.235518 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.235500 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2047fac-4703-46be-9363-175e876be523-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:02.235561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.235520 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w52qk\" (UniqueName: \"kubernetes.io/projected/e2047fac-4703-46be-9363-175e876be523-kube-api-access-w52qk\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:02.247180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.247157 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds"] Sep 10 14:48:02.250065 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.250036 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:02.251247 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.251219 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5c43af-9994f6647-5l9ds"] Sep 10 14:48:02.560361 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.560319 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:02.760699 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.760666 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a36fd75d-4b3c-4671-a810-89d846acc932" path="/var/lib/kubelet/pods/a36fd75d-4b3c-4671-a810-89d846acc932/volumes" Sep 10 14:48:02.761113 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:02.761092 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2047fac-4703-46be-9363-175e876be523" path="/var/lib/kubelet/pods/e2047fac-4703-46be-9363-175e876be523/volumes" Sep 10 14:48:06.608442 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:06.608365 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:48:08.342462 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.342425 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:08.412726 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.412690 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:08.604171 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.604084 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:08.928003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.927970 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5"] Sep 10 14:48:08.928367 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.928353 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a36fd75d-4b3c-4671-a810-89d846acc932" containerName="revoke-keys" Sep 10 14:48:08.928411 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.928368 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a36fd75d-4b3c-4671-a810-89d846acc932" containerName="revoke-keys" Sep 10 14:48:08.928411 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.928378 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2047fac-4703-46be-9363-175e876be523" containerName="payload-processing" Sep 10 14:48:08.928411 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.928384 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2047fac-4703-46be-9363-175e876be523" containerName="payload-processing" Sep 10 14:48:08.928527 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.928464 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a36fd75d-4b3c-4671-a810-89d846acc932" containerName="revoke-keys" Sep 10 14:48:08.928527 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.928476 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2047fac-4703-46be-9363-175e876be523" containerName="payload-processing" Sep 10 14:48:08.932956 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.932941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:08.935563 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.935542 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:48:08.935683 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.935586 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:48:08.939273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.939249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5"] Sep 10 14:48:08.994758 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:08.994724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3de6589-e112-49b3-9850-6cea591f833d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5\" (UID: \"f3de6589-e112-49b3-9850-6cea591f833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:09.023445 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:09.023418 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:09.095883 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:09.095848 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3de6589-e112-49b3-9850-6cea591f833d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5\" (UID: \"f3de6589-e112-49b3-9850-6cea591f833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:09.096471 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:09.096453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3de6589-e112-49b3-9850-6cea591f833d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5\" (UID: \"f3de6589-e112-49b3-9850-6cea591f833d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:09.244144 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:09.244062 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:09.514568 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:09.514539 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5"] Sep 10 14:48:09.517022 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:48:09.516995 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3de6589_e112_49b3_9850_6cea591f833d.slice/crio-676cb38d940899af61f5043314d645214a20b56546d92b94ea014d0ac2f3b780 WatchSource:0}: Error finding container 676cb38d940899af61f5043314d645214a20b56546d92b94ea014d0ac2f3b780: Status 404 returned error can't find the container with id 676cb38d940899af61f5043314d645214a20b56546d92b94ea014d0ac2f3b780 Sep 10 14:48:10.269738 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:10.269707 2576 generic.go:356] "Generic (PLEG): container finished" podID="f3de6589-e112-49b3-9850-6cea591f833d" containerID="f07026b64e8feb12b00db524a93149d8a3cc90d03e903f84d23124dd9e7c74e6" exitCode=0 Sep 10 14:48:10.269893 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:10.269814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" event={"ID":"f3de6589-e112-49b3-9850-6cea591f833d","Type":"ContainerDied","Data":"f07026b64e8feb12b00db524a93149d8a3cc90d03e903f84d23124dd9e7c74e6"} Sep 10 14:48:10.269893 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:10.269859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" event={"ID":"f3de6589-e112-49b3-9850-6cea591f833d","Type":"ContainerStarted","Data":"676cb38d940899af61f5043314d645214a20b56546d92b94ea014d0ac2f3b780"} Sep 10 14:48:11.404879 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:11.404856 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:11.518704 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:11.518675 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3de6589-e112-49b3-9850-6cea591f833d-maas-api-service-ca\") pod \"f3de6589-e112-49b3-9850-6cea591f833d\" (UID: \"f3de6589-e112-49b3-9850-6cea591f833d\") " Sep 10 14:48:11.519035 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:11.519014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3de6589-e112-49b3-9850-6cea591f833d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3de6589-e112-49b3-9850-6cea591f833d" (UID: "f3de6589-e112-49b3-9850-6cea591f833d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:48:11.620295 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:11.620219 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3de6589-e112-49b3-9850-6cea591f833d-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:12.280368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:12.280319 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" event={"ID":"f3de6589-e112-49b3-9850-6cea591f833d","Type":"ContainerDied","Data":"676cb38d940899af61f5043314d645214a20b56546d92b94ea014d0ac2f3b780"} Sep 10 14:48:12.280368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:12.280347 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5" Sep 10 14:48:12.280368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:12.280373 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="676cb38d940899af61f5043314d645214a20b56546d92b94ea014d0ac2f3b780" Sep 10 14:48:19.339141 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.339110 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5"] Sep 10 14:48:19.345518 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.345484 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:19.345989 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.345969 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7bf16f-29817525-mxxq5"] Sep 10 14:48:19.366821 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.366775 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf"] Sep 10 14:48:19.367107 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.367074 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" podUID="a1991246-c7c1-41f5-b6af-d0b5d55db3fc" containerName="payload-processing" containerID="cri-o://ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f" gracePeriod=30 Sep 10 14:48:19.507041 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.507006 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5"] Sep 10 14:48:19.509046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.509022 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7bf16f-vgmt5"] Sep 10 14:48:19.622668 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.622647 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:48:19.622906 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.622881 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:19.690968 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.690935 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjkq8\" (UniqueName: \"kubernetes.io/projected/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-kube-api-access-fjkq8\") pod \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " Sep 10 14:48:19.691119 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.691110 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-plugins-config-volume\") pod \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\" (UID: \"a1991246-c7c1-41f5-b6af-d0b5d55db3fc\") " Sep 10 14:48:19.691477 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.691452 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1991246-c7c1-41f5-b6af-d0b5d55db3fc" (UID: "a1991246-c7c1-41f5-b6af-d0b5d55db3fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:48:19.693069 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.693045 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-kube-api-access-fjkq8" (OuterVolumeSpecName: "kube-api-access-fjkq8") pod "a1991246-c7c1-41f5-b6af-d0b5d55db3fc" (UID: "a1991246-c7c1-41f5-b6af-d0b5d55db3fc"). InnerVolumeSpecName "kube-api-access-fjkq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:48:19.724425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.724395 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:19.792248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.792219 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:19.792248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.792244 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjkq8\" (UniqueName: \"kubernetes.io/projected/a1991246-c7c1-41f5-b6af-d0b5d55db3fc-kube-api-access-fjkq8\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:19.883767 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:19.883731 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:20.323050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.322963 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1991246-c7c1-41f5-b6af-d0b5d55db3fc" containerID="ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f" exitCode=0 Sep 10 14:48:20.323050 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.323009 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" Sep 10 14:48:20.323226 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.323052 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" event={"ID":"a1991246-c7c1-41f5-b6af-d0b5d55db3fc","Type":"ContainerDied","Data":"ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f"} Sep 10 14:48:20.323226 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.323104 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf" event={"ID":"a1991246-c7c1-41f5-b6af-d0b5d55db3fc","Type":"ContainerDied","Data":"86ebe4a48e2641899aed23e1b7b4b7467d0b5f90f2040620c32605217565f0c0"} Sep 10 14:48:20.323226 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.323126 2576 scope.go:117] "RemoveContainer" containerID="ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f" Sep 10 14:48:20.331677 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.331660 2576 scope.go:117] "RemoveContainer" containerID="ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f" Sep 10 14:48:20.331945 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:48:20.331927 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f\": container with ID starting with ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f not found: ID does not exist" containerID="ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f" Sep 10 14:48:20.332027 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.331952 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f"} err="failed to get container status \"ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f\": rpc error: code = NotFound desc = could not find container \"ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f\": container with ID starting with ac618716bd301f79e9d77cd862054e24b9223d379b7678826c97903571ac9a9f not found: ID does not exist" Sep 10 14:48:20.343464 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.343443 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf"] Sep 10 14:48:20.345588 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.345563 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-7bf16f-5c665dfdc-npdbf"] Sep 10 14:48:20.761376 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.761344 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1991246-c7c1-41f5-b6af-d0b5d55db3fc" path="/var/lib/kubelet/pods/a1991246-c7c1-41f5-b6af-d0b5d55db3fc/volumes" Sep 10 14:48:20.761645 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.761631 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed2ec44-b546-45fb-959f-ba21549a3532" path="/var/lib/kubelet/pods/aed2ec44-b546-45fb-959f-ba21549a3532/volumes" Sep 10 14:48:20.761946 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:20.761932 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3de6589-e112-49b3-9850-6cea591f833d" path="/var/lib/kubelet/pods/f3de6589-e112-49b3-9850-6cea591f833d/volumes" Sep 10 14:48:21.872070 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:21.872027 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:21.972021 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:21.971991 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:22.108389 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:22.108355 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:48:25.887269 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:25.887234 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:25.987593 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:25.987560 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:26.151046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.150950 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:26.496785 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.496704 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l"] Sep 10 14:48:26.497172 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.497153 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1991246-c7c1-41f5-b6af-d0b5d55db3fc" containerName="payload-processing" Sep 10 14:48:26.497172 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.497173 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1991246-c7c1-41f5-b6af-d0b5d55db3fc" containerName="payload-processing" Sep 10 14:48:26.497311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.497193 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3de6589-e112-49b3-9850-6cea591f833d" containerName="revoke-keys" Sep 10 14:48:26.497311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.497199 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3de6589-e112-49b3-9850-6cea591f833d" containerName="revoke-keys" Sep 10 14:48:26.497311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.497272 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1991246-c7c1-41f5-b6af-d0b5d55db3fc" containerName="payload-processing" Sep 10 14:48:26.497311 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.497279 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3de6589-e112-49b3-9850-6cea591f833d" containerName="revoke-keys" Sep 10 14:48:26.501504 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.501488 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:26.504410 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.504391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 14:48:26.504490 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.504466 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwr7b\"" Sep 10 14:48:26.508911 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.508576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l"] Sep 10 14:48:26.597945 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.597912 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:26.655207 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.655176 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5957d58a-512f-44e7-b9e0-41d82a34bb88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l\" (UID: \"5957d58a-512f-44e7-b9e0-41d82a34bb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:26.756384 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.756298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5957d58a-512f-44e7-b9e0-41d82a34bb88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l\" (UID: \"5957d58a-512f-44e7-b9e0-41d82a34bb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:26.756901 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.756881 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5957d58a-512f-44e7-b9e0-41d82a34bb88-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l\" (UID: \"5957d58a-512f-44e7-b9e0-41d82a34bb88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:26.812597 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:26.812564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:27.082947 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:27.082925 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l"] Sep 10 14:48:27.085305 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:48:27.085279 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5957d58a_512f_44e7_b9e0_41d82a34bb88.slice/crio-6fd0e60299818c026a9399d45d4e0dc6d1380130bc71e254d53e9072c83756d2 WatchSource:0}: Error finding container 6fd0e60299818c026a9399d45d4e0dc6d1380130bc71e254d53e9072c83756d2: Status 404 returned error can't find the container with id 6fd0e60299818c026a9399d45d4e0dc6d1380130bc71e254d53e9072c83756d2 Sep 10 14:48:27.363273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:27.363196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" event={"ID":"5957d58a-512f-44e7-b9e0-41d82a34bb88","Type":"ContainerStarted","Data":"473e737a8c7f34c1f96b6a98a50e739820e2fb4513141a02ee4225fe150c4077"} Sep 10 14:48:27.363273 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:27.363249 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" event={"ID":"5957d58a-512f-44e7-b9e0-41d82a34bb88","Type":"ContainerStarted","Data":"6fd0e60299818c026a9399d45d4e0dc6d1380130bc71e254d53e9072c83756d2"} Sep 10 14:48:27.377709 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:27.377652 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" podStartSLOduration=1.377636093 podStartE2EDuration="1.377636093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:48:27.376814636 +0000 UTC m=+1725.230723313" watchObservedRunningTime="2026-09-10 14:48:27.377636093 +0000 UTC m=+1725.231544768" Sep 10 14:48:28.369483 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:28.369452 2576 generic.go:356] "Generic (PLEG): container finished" podID="5957d58a-512f-44e7-b9e0-41d82a34bb88" containerID="473e737a8c7f34c1f96b6a98a50e739820e2fb4513141a02ee4225fe150c4077" exitCode=0 Sep 10 14:48:28.369994 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:28.369493 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" event={"ID":"5957d58a-512f-44e7-b9e0-41d82a34bb88","Type":"ContainerDied","Data":"473e737a8c7f34c1f96b6a98a50e739820e2fb4513141a02ee4225fe150c4077"} Sep 10 14:48:29.504197 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:29.504167 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:29.581665 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:29.581633 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5957d58a-512f-44e7-b9e0-41d82a34bb88-maas-api-service-ca\") pod \"5957d58a-512f-44e7-b9e0-41d82a34bb88\" (UID: \"5957d58a-512f-44e7-b9e0-41d82a34bb88\") " Sep 10 14:48:29.581970 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:29.581947 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5957d58a-512f-44e7-b9e0-41d82a34bb88-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5957d58a-512f-44e7-b9e0-41d82a34bb88" (UID: "5957d58a-512f-44e7-b9e0-41d82a34bb88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:48:29.682808 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:29.682760 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5957d58a-512f-44e7-b9e0-41d82a34bb88-maas-api-service-ca\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:30.380941 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:30.380898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" event={"ID":"5957d58a-512f-44e7-b9e0-41d82a34bb88","Type":"ContainerDied","Data":"6fd0e60299818c026a9399d45d4e0dc6d1380130bc71e254d53e9072c83756d2"} Sep 10 14:48:30.381120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:30.380956 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd0e60299818c026a9399d45d4e0dc6d1380130bc71e254d53e9072c83756d2" Sep 10 14:48:30.381120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:30.380921 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l" Sep 10 14:48:36.910989 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:36.910949 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s"] Sep 10 14:48:36.915490 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:36.915462 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-59dc59-29817525-jht6s"] Sep 10 14:48:36.921849 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:36.921822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:36.940516 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:36.940484 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c"] Sep 10 14:48:36.940749 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:36.940720 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" podUID="12896b5b-aaca-48de-aceb-7246546660f8" containerName="payload-processing" containerID="cri-o://9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07" gracePeriod=30 Sep 10 14:48:37.023064 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.023020 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:37.121419 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.121383 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:37.126416 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.126390 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l"] Sep 10 14:48:37.129156 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.129132 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-59dc59-h8d8l"] Sep 10 14:48:37.192482 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.192458 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:48:37.215897 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.215854 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:37.340561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.340528 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjn28\" (UniqueName: \"kubernetes.io/projected/12896b5b-aaca-48de-aceb-7246546660f8-kube-api-access-gjn28\") pod \"12896b5b-aaca-48de-aceb-7246546660f8\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " Sep 10 14:48:37.340726 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.340652 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12896b5b-aaca-48de-aceb-7246546660f8-plugins-config-volume\") pod \"12896b5b-aaca-48de-aceb-7246546660f8\" (UID: \"12896b5b-aaca-48de-aceb-7246546660f8\") " Sep 10 14:48:37.341036 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.341014 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12896b5b-aaca-48de-aceb-7246546660f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12896b5b-aaca-48de-aceb-7246546660f8" (UID: "12896b5b-aaca-48de-aceb-7246546660f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 14:48:37.342603 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.342576 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12896b5b-aaca-48de-aceb-7246546660f8-kube-api-access-gjn28" (OuterVolumeSpecName: "kube-api-access-gjn28") pod "12896b5b-aaca-48de-aceb-7246546660f8" (UID: "12896b5b-aaca-48de-aceb-7246546660f8"). InnerVolumeSpecName "kube-api-access-gjn28". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:48:37.418543 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.418510 2576 generic.go:356] "Generic (PLEG): container finished" podID="12896b5b-aaca-48de-aceb-7246546660f8" containerID="9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07" exitCode=0 Sep 10 14:48:37.418717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.418562 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" Sep 10 14:48:37.418717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.418595 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" event={"ID":"12896b5b-aaca-48de-aceb-7246546660f8","Type":"ContainerDied","Data":"9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07"} Sep 10 14:48:37.418717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.418667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c" event={"ID":"12896b5b-aaca-48de-aceb-7246546660f8","Type":"ContainerDied","Data":"3cbacc78428c836007932440b2a677ed827e7d6e0563f58dcad5b408d663901f"} Sep 10 14:48:37.418717 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.418689 2576 scope.go:117] "RemoveContainer" containerID="9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07" Sep 10 14:48:37.427985 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.427968 2576 scope.go:117] "RemoveContainer" containerID="9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07" Sep 10 14:48:37.428246 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:48:37.428228 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07\": container with ID starting with 9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07 not found: ID does not exist" containerID="9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07" Sep 10 14:48:37.428319 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.428252 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07"} err="failed to get container status \"9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07\": rpc error: code = NotFound desc = could not find container \"9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07\": container with ID starting with 9d5e1288b4836a60ae7da565d6f3309e9a99ffd46557f73888c3af13c55a7b07 not found: ID does not exist" Sep 10 14:48:37.440177 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.440157 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c"] Sep 10 14:48:37.442046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.442025 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12896b5b-aaca-48de-aceb-7246546660f8-plugins-config-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:37.442046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.442046 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjn28\" (UniqueName: \"kubernetes.io/projected/12896b5b-aaca-48de-aceb-7246546660f8-kube-api-access-gjn28\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:48:37.442290 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.442272 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-59dc59-dc7f8ff87-x7w5c"] Sep 10 14:48:37.542357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:37.542327 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:38.760439 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:38.760408 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12896b5b-aaca-48de-aceb-7246546660f8" path="/var/lib/kubelet/pods/12896b5b-aaca-48de-aceb-7246546660f8/volumes" Sep 10 14:48:38.760886 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:38.760760 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5957d58a-512f-44e7-b9e0-41d82a34bb88" path="/var/lib/kubelet/pods/5957d58a-512f-44e7-b9e0-41d82a34bb88/volumes" Sep 10 14:48:38.761046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:38.761031 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="606f1938-4aa0-486a-a890-9d77f8d3a5a8" path="/var/lib/kubelet/pods/606f1938-4aa0-486a-a890-9d77f8d3a5a8/volumes" Sep 10 14:48:39.416885 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:39.416846 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:48:42.891003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:42.890968 2576 scope.go:117] "RemoveContainer" containerID="90996934b56568eb0790e5223ae1138cdb85c439184c89c056c16f1468bebaa2" Sep 10 14:48:43.460094 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:43.460059 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:43.571882 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:43.571849 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:43.712688 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:43.712594 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:43.871685 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:43.871652 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:43.939120 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:43.939092 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:48:48.218588 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:48:48.218543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:49:08.826154 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:08.826118 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:42.674110 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:42.674085 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:49:42.679905 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:42.679881 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:49:42.920667 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:42.920631 2576 scope.go:117] "RemoveContainer" containerID="283cb7f0a44f281d11ad75b96742b9aa9f8178912f102c19140b6400968eec0f" Sep 10 14:49:42.928601 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:42.928578 2576 scope.go:117] "RemoveContainer" containerID="ab90703466e16a4a9dbca2c5a384e6168d5c68caaec72300fc13199007c27b02" Sep 10 14:49:44.422909 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:44.422877 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:48.748454 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:48.748419 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:49:48.849390 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:49:48.849352 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:03.720943 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:03.720888 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:09.746580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:09.746539 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:10.721297 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:10.721259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:50:11.727357 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:11.727317 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:13.176443 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:13.176399 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:50:35.213271 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:35.213237 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:50:42.815190 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:42.815149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:50:49.816473 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:49.816435 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:50:56.011421 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:50:56.011376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:51:09.511605 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:09.511524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:51:17.612453 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:17.612417 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:51:24.588843 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.588806 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c"] Sep 10 14:51:24.589382 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.589028 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" podUID="7eaccd13-9f96-447c-a418-82b10db5750e" containerName="manager" containerID="cri-o://ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c" gracePeriod=10 Sep 10 14:51:24.619125 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.619091 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" podUID="7eaccd13-9f96-447c-a418-82b10db5750e" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.31:8081/readyz\": dial tcp 10.134.0.31:8081: connect: connection refused" Sep 10 14:51:24.837673 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.837647 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:51:24.958966 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.958931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eaccd13-9f96-447c-a418-82b10db5750e-extensions-socket-volume\") pod \"7eaccd13-9f96-447c-a418-82b10db5750e\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " Sep 10 14:51:24.959128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.959039 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwz45\" (UniqueName: \"kubernetes.io/projected/7eaccd13-9f96-447c-a418-82b10db5750e-kube-api-access-lwz45\") pod \"7eaccd13-9f96-447c-a418-82b10db5750e\" (UID: \"7eaccd13-9f96-447c-a418-82b10db5750e\") " Sep 10 14:51:24.959342 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.959315 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7eaccd13-9f96-447c-a418-82b10db5750e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7eaccd13-9f96-447c-a418-82b10db5750e" (UID: "7eaccd13-9f96-447c-a418-82b10db5750e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 14:51:24.961015 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:24.960993 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eaccd13-9f96-447c-a418-82b10db5750e-kube-api-access-lwz45" (OuterVolumeSpecName: "kube-api-access-lwz45") pod "7eaccd13-9f96-447c-a418-82b10db5750e" (UID: "7eaccd13-9f96-447c-a418-82b10db5750e"). InnerVolumeSpecName "kube-api-access-lwz45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 14:51:25.059811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.059746 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7eaccd13-9f96-447c-a418-82b10db5750e-extensions-socket-volume\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:51:25.059811 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.059812 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwz45\" (UniqueName: \"kubernetes.io/projected/7eaccd13-9f96-447c-a418-82b10db5750e-kube-api-access-lwz45\") on node \"ip-10-0-135-236.ec2.internal\" DevicePath \"\"" Sep 10 14:51:25.236047 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.235954 2576 generic.go:356] "Generic (PLEG): container finished" podID="7eaccd13-9f96-447c-a418-82b10db5750e" containerID="ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c" exitCode=0 Sep 10 14:51:25.236047 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.235998 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" Sep 10 14:51:25.236248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.236040 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" event={"ID":"7eaccd13-9f96-447c-a418-82b10db5750e","Type":"ContainerDied","Data":"ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c"} Sep 10 14:51:25.236248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.236090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c" event={"ID":"7eaccd13-9f96-447c-a418-82b10db5750e","Type":"ContainerDied","Data":"eb1ef1e330210cb45da4d2ad10f7d5366f25d824dcdcf483d06cbf47833187fb"} Sep 10 14:51:25.236248 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.236113 2576 scope.go:117] "RemoveContainer" containerID="ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c" Sep 10 14:51:25.245679 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.245663 2576 scope.go:117] "RemoveContainer" containerID="ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c" Sep 10 14:51:25.245956 ip-10-0-135-236 kubenswrapper[2576]: E0910 14:51:25.245935 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c\": container with ID starting with ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c not found: ID does not exist" containerID="ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c" Sep 10 14:51:25.246007 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.245964 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c"} err="failed to get container status \"ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c\": rpc error: code = NotFound desc = could not find container \"ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c\": container with ID starting with ee7243323bcc6bfe180311c323312765959372df0b3b7d9742b56541bb34d41c not found: ID does not exist" Sep 10 14:51:25.257399 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.257375 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c"] Sep 10 14:51:25.261238 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:25.261213 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kql7c"] Sep 10 14:51:26.760336 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:26.760305 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7eaccd13-9f96-447c-a418-82b10db5750e" path="/var/lib/kubelet/pods/7eaccd13-9f96-447c-a418-82b10db5750e/volumes" Sep 10 14:51:42.957254 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:42.957221 2576 scope.go:117] "RemoveContainer" containerID="1a5d4442f6f572c0f0d825e824a79125520482cd271a844239231a07ba033f16" Sep 10 14:51:42.967078 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:51:42.967057 2576 scope.go:117] "RemoveContainer" containerID="e709e56edaf2fac47ca081f38c67dc038be80a25540df5c9befa15200d1c40af" Sep 10 14:52:25.276580 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276546 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd"] Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276891 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5957d58a-512f-44e7-b9e0-41d82a34bb88" containerName="revoke-keys" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276903 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5957d58a-512f-44e7-b9e0-41d82a34bb88" containerName="revoke-keys" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276914 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7eaccd13-9f96-447c-a418-82b10db5750e" containerName="manager" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276919 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7eaccd13-9f96-447c-a418-82b10db5750e" containerName="manager" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276933 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12896b5b-aaca-48de-aceb-7246546660f8" containerName="payload-processing" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276940 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12896b5b-aaca-48de-aceb-7246546660f8" containerName="payload-processing" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276986 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12896b5b-aaca-48de-aceb-7246546660f8" containerName="payload-processing" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.276995 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5957d58a-512f-44e7-b9e0-41d82a34bb88" containerName="revoke-keys" Sep 10 14:52:25.277049 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.277002 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7eaccd13-9f96-447c-a418-82b10db5750e" containerName="manager" Sep 10 14:52:25.279889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.279872 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.282583 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.282563 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zk7bh\"" Sep 10 14:52:25.293889 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.293863 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd"] Sep 10 14:52:25.396809 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.396760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e093947-243a-4f75-a1cf-432700419a10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ckcxd\" (UID: \"1e093947-243a-4f75-a1cf-432700419a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.396999 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.396848 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvssq\" (UniqueName: \"kubernetes.io/projected/1e093947-243a-4f75-a1cf-432700419a10-kube-api-access-qvssq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ckcxd\" (UID: \"1e093947-243a-4f75-a1cf-432700419a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.498046 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.498005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvssq\" (UniqueName: \"kubernetes.io/projected/1e093947-243a-4f75-a1cf-432700419a10-kube-api-access-qvssq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ckcxd\" (UID: \"1e093947-243a-4f75-a1cf-432700419a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.498227 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.498121 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e093947-243a-4f75-a1cf-432700419a10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ckcxd\" (UID: \"1e093947-243a-4f75-a1cf-432700419a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.498494 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.498476 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e093947-243a-4f75-a1cf-432700419a10-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ckcxd\" (UID: \"1e093947-243a-4f75-a1cf-432700419a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.509579 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.509544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvssq\" (UniqueName: \"kubernetes.io/projected/1e093947-243a-4f75-a1cf-432700419a10-kube-api-access-qvssq\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ckcxd\" (UID: \"1e093947-243a-4f75-a1cf-432700419a10\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.590364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.590276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:25.867744 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:25.867715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd"] Sep 10 14:52:25.870655 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:52:25.870625 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e093947_243a_4f75_a1cf_432700419a10.slice/crio-1bfc61b848c0e9b0202062f46142616e2b791e155f593cc23c565b654b7ee240 WatchSource:0}: Error finding container 1bfc61b848c0e9b0202062f46142616e2b791e155f593cc23c565b654b7ee240: Status 404 returned error can't find the container with id 1bfc61b848c0e9b0202062f46142616e2b791e155f593cc23c565b654b7ee240 Sep 10 14:52:26.534813 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:26.534752 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" event={"ID":"1e093947-243a-4f75-a1cf-432700419a10","Type":"ContainerStarted","Data":"51832d82820eea676aac3dc7dfcb6b7d2de98db6cb72bc16c6fa666230b55d40"} Sep 10 14:52:26.535294 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:26.534827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" event={"ID":"1e093947-243a-4f75-a1cf-432700419a10","Type":"ContainerStarted","Data":"1bfc61b848c0e9b0202062f46142616e2b791e155f593cc23c565b654b7ee240"} Sep 10 14:52:26.535294 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:26.534848 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:26.570667 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:26.570601 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" podStartSLOduration=1.570579554 podStartE2EDuration="1.570579554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 14:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:52:26.569101314 +0000 UTC m=+1964.423009980" watchObservedRunningTime="2026-09-10 14:52:26.570579554 +0000 UTC m=+1964.424488232" Sep 10 14:52:37.542488 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:37.542412 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ckcxd" Sep 10 14:52:42.991065 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:42.991029 2576 scope.go:117] "RemoveContainer" containerID="14c4190b771a189b3e12f43d0123c0934ce5bd011ebcdbc60dd006e7e95faf3d" Sep 10 14:52:46.910857 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:46.910819 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:52:55.105447 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:52:55.105412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:53:00.909599 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:00.909549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:53:25.046614 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:25.046582 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:27.416232 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:27.416194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:53:43.009180 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:43.009148 2576 scope.go:117] "RemoveContainer" containerID="1fabff5d8f88b849c5fafc0d2cf64cefc09983cc987cfa75c63753c96af0a024" Sep 10 14:53:46.245706 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:46.245674 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:53:49.105128 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:49.105091 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:53:52.257364 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:53:52.257305 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:04.815664 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:04.815630 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:06.544368 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:06.544322 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:06.817378 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:06.817301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:15.518880 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:15.518848 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:22.712086 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:22.712048 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:27.687363 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:27.687330 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:27.784753 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:27.784725 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:54:29.414744 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:29.414709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:37.211906 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:37.211868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:42.700672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:42.700644 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:54:42.707936 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:42.707915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:54:43.027627 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:43.027549 2576 scope.go:117] "RemoveContainer" containerID="473e737a8c7f34c1f96b6a98a50e739820e2fb4513141a02ee4225fe150c4077" Sep 10 14:54:43.035561 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:43.035544 2576 scope.go:117] "RemoveContainer" containerID="f07026b64e8feb12b00db524a93149d8a3cc90d03e903f84d23124dd9e7c74e6" Sep 10 14:54:43.042586 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:43.042569 2576 scope.go:117] "RemoveContainer" containerID="9cd9cb8a03b10a0467aaeb4cf0ab057bfb9a8dba6fdcd313a4a189868d36393d" Sep 10 14:54:43.802160 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:43.802115 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:49.710773 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:49.710730 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:56.199264 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:56.199229 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-x9gtg"] Sep 10 14:54:59.492301 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:54:59.492260 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d445769f-zlsf6_369aaedc-43e4-4994-9bfe-7fd9e95b1f69/authorino/0.log" Sep 10 14:55:04.623779 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:04.623736 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ggb62_fe5a37b4-2adc-41a4-8dee-4deac62d1414/manager/0.log" Sep 10 14:55:04.735831 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:04.735782 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mc25h_5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb/manager/0.log" Sep 10 14:55:04.994002 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:04.993972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-phsrl_2fbbd3d3-18af-465f-9ce7-323d095d83cd/server/0.log" Sep 10 14:55:05.458552 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:05.458520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-vlqwk_b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92/copy-manifests/0.log" Sep 10 14:55:05.470817 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:05.470763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-vlqwk_b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92/manager/0.log" Sep 10 14:55:37.067055 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:37.066976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d445769f-zlsf6_369aaedc-43e4-4994-9bfe-7fd9e95b1f69/authorino/0.log" Sep 10 14:55:37.181378 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:37.181346 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-mft7v_6329f60d-daeb-4fa8-81e9-493efb21f2ee/manager/0.log" Sep 10 14:55:37.298681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:37.298647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rvp7f_e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf/manager/0.log" Sep 10 14:55:37.538871 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:37.538840 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ckcxd_1e093947-243a-4f75-a1cf-432700419a10/manager/0.log" Sep 10 14:55:37.653221 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:37.653190 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-x9gtg_4f220e00-79a0-4497-bc8c-aaf8865eeedd/limitador/0.log" Sep 10 14:55:37.781605 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:37.781567 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dgdc4_ad0fc8cb-763d-4bbd-9865-168871b6286c/manager/0.log" Sep 10 14:55:38.151584 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:38.151540 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv_131c2791-30c9-480d-9ac2-19739506c833/istio-proxy/0.log" Sep 10 14:55:38.378706 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:38.378674 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-2ctcp_c25c4402-a1d2-4635-8806-a2a598642bfe/kube-auth-proxy/0.log" Sep 10 14:55:39.400419 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:39.400392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kkrqf_ba9aa1d6-64d9-4634-a30f-575d8d97298f/main/0.log" Sep 10 14:55:39.623959 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:39.623930 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-r7z8k_533f97d3-4209-4c81-b8b1-f0984f175644/main/0.log" Sep 10 14:55:39.835955 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:55:39.835875 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bf2tg_5893093a-c52e-40a5-9c5f-8e681e7c1956/main/0.log" Sep 10 14:56:04.491584 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:04.491545 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:05.054856 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:05.054822 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:05.226567 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:05.226524 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:05.326539 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:05.326454 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" limit=1024 Sep 10 14:56:07.849888 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:07.849857 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9pl2k_7fa850c7-5397-4be3-80a9-2afab7eca821/global-pull-secret-syncer/0.log" Sep 10 14:56:07.927489 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:07.927454 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2m7lm_87136eb6-b040-4611-9150-8308af770a4b/konnectivity-agent/0.log" Sep 10 14:56:08.016973 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:08.016940 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-236.ec2.internal_72e1c97d5f33b9f41328e5f5e35a0ce0/haproxy/0.log" Sep 10 14:56:12.742175 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:12.742135 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75d445769f-zlsf6_369aaedc-43e4-4994-9bfe-7fd9e95b1f69/authorino/0.log" Sep 10 14:56:12.765452 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:12.765424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-mft7v_6329f60d-daeb-4fa8-81e9-493efb21f2ee/manager/0.log" Sep 10 14:56:12.791155 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:12.791124 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rvp7f_e1e0cc5d-bd53-43ce-bae9-75bce3a41bbf/manager/0.log" Sep 10 14:56:12.891942 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:12.891907 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ckcxd_1e093947-243a-4f75-a1cf-432700419a10/manager/0.log" Sep 10 14:56:12.909643 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:12.909615 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-x9gtg_4f220e00-79a0-4497-bc8c-aaf8865eeedd/limitador/0.log" Sep 10 14:56:12.968658 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:12.968622 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dgdc4_ad0fc8cb-763d-4bbd-9865-168871b6286c/manager/0.log" Sep 10 14:56:14.727531 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:14.727478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-688cf66c9d-v529b_b00406b1-e4ce-4a29-99d1-01fbaed628ad/metrics-server/0.log" Sep 10 14:56:14.947687 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:14.947655 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm8fz_f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd/node-exporter/0.log" Sep 10 14:56:14.971682 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:14.971651 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm8fz_f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd/kube-rbac-proxy/0.log" Sep 10 14:56:14.987618 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:14.987555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sm8fz_f48ad3fd-bdfe-4b17-9ef8-eedbbcfc08cd/init-textfile/0.log" Sep 10 14:56:18.096309 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.096272 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-556fc4b877-89zkr_d3a0c1fe-aa59-4b47-a0f5-61c9733c9862/console/0.log" Sep 10 14:56:18.249157 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.249123 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx"] Sep 10 14:56:18.252455 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.252434 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.255139 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.255117 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4xhnq\"/\"openshift-service-ca.crt\"" Sep 10 14:56:18.255263 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.255244 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4xhnq\"/\"kube-root-ca.crt\"" Sep 10 14:56:18.256490 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.256470 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4xhnq\"/\"default-dockercfg-24sbf\"" Sep 10 14:56:18.260425 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.260402 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx"] Sep 10 14:56:18.290055 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.290023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-proc\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.290222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.290079 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-sys\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.290222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.290102 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-lib-modules\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.290222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.290125 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v4gs\" (UniqueName: \"kubernetes.io/projected/9e61d3bc-4471-44e6-b593-186aaef07f4f-kube-api-access-5v4gs\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.290222 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.290157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-podres\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391494 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-podres\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391543 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-proc\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-sys\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-lib-modules\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v4gs\" (UniqueName: \"kubernetes.io/projected/9e61d3bc-4471-44e6-b593-186aaef07f4f-kube-api-access-5v4gs\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-podres\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391681 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-proc\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391926 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-sys\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.391926 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.391762 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e61d3bc-4471-44e6-b593-186aaef07f4f-lib-modules\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.400416 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.400393 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v4gs\" (UniqueName: \"kubernetes.io/projected/9e61d3bc-4471-44e6-b593-186aaef07f4f-kube-api-access-5v4gs\") pod \"perf-node-gather-daemonset-hh2dx\" (UID: \"9e61d3bc-4471-44e6-b593-186aaef07f4f\") " pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.563672 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.563633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:18.844742 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:18.844713 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx"] Sep 10 14:56:18.846464 ip-10-0-135-236 kubenswrapper[2576]: W0910 14:56:18.846430 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e61d3bc_4471_44e6_b593_186aaef07f4f.slice/crio-e8fe27c8012986581dbdcb171d6f22bf6303aaae07e49d61d8fca9d006bc083c WatchSource:0}: Error finding container e8fe27c8012986581dbdcb171d6f22bf6303aaae07e49d61d8fca9d006bc083c: Status 404 returned error can't find the container with id e8fe27c8012986581dbdcb171d6f22bf6303aaae07e49d61d8fca9d006bc083c Sep 10 14:56:19.627118 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:19.627079 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" event={"ID":"9e61d3bc-4471-44e6-b593-186aaef07f4f","Type":"ContainerStarted","Data":"d913620421a7a924bc34d4b755559696e143d401795b66e2310d564cf66788da"} Sep 10 14:56:19.627505 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:19.627142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" event={"ID":"9e61d3bc-4471-44e6-b593-186aaef07f4f","Type":"ContainerStarted","Data":"e8fe27c8012986581dbdcb171d6f22bf6303aaae07e49d61d8fca9d006bc083c"} Sep 10 14:56:19.627505 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:19.627196 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:19.643594 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:19.643548 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" podStartSLOduration=1.6435333220000001 podStartE2EDuration="1.643533322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 14:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 14:56:19.64313486 +0000 UTC m=+2197.497043536" watchObservedRunningTime="2026-09-10 14:56:19.643533322 +0000 UTC m=+2197.497441998" Sep 10 14:56:20.042219 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:20.042109 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25qhx_649a3eab-a879-4a4b-9d33-5e3d621a5953/dns/0.log" Sep 10 14:56:20.061255 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:20.061234 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-25qhx_649a3eab-a879-4a4b-9d33-5e3d621a5953/kube-rbac-proxy/0.log" Sep 10 14:56:20.186914 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:20.186880 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8998p_26d0251c-9cab-48df-bd56-a6724f77f293/dns-node-resolver/0.log" Sep 10 14:56:20.772147 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:20.772113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-655f9c8b9f-9vgf4_684530a2-e166-41a2-8f5c-728e8eaad971/registry/0.log" Sep 10 14:56:20.830279 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:20.830244 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-pwhpf_a1aa96c3-14fa-4594-ba74-97b26ff67379/node-ca/0.log" Sep 10 14:56:21.698037 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:21.698002 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv_131c2791-30c9-480d-9ac2-19739506c833/istio-proxy/0.log" Sep 10 14:56:21.935569 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:21.935535 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-2ctcp_c25c4402-a1d2-4635-8806-a2a598642bfe/kube-auth-proxy/0.log" Sep 10 14:56:23.007776 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:23.007744 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gb9tp_5b8bb0f6-c6b4-4a22-b678-8f2b65d28b40/serve-healthcheck-canary/0.log" Sep 10 14:56:23.521495 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:23.521466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k5ngq_ca9f5f01-a250-4c9c-9d20-6bf062b43632/kube-rbac-proxy/0.log" Sep 10 14:56:23.537823 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:23.537779 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k5ngq_ca9f5f01-a250-4c9c-9d20-6bf062b43632/exporter/0.log" Sep 10 14:56:23.553587 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:23.553555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k5ngq_ca9f5f01-a250-4c9c-9d20-6bf062b43632/extractor/0.log" Sep 10 14:56:25.642423 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:25.642396 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4xhnq/perf-node-gather-daemonset-hh2dx" Sep 10 14:56:25.688003 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:25.687971 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-ggb62_fe5a37b4-2adc-41a4-8dee-4deac62d1414/manager/0.log" Sep 10 14:56:25.857927 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:25.857889 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mc25h_5c6048e5-38d1-4e1f-9ba2-50aa6a5e85cb/manager/0.log" Sep 10 14:56:25.927936 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:25.927858 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-phsrl_2fbbd3d3-18af-465f-9ce7-323d095d83cd/server/0.log" Sep 10 14:56:26.143279 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:26.143246 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-vlqwk_b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92/manager/0.log" Sep 10 14:56:26.158151 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:26.158122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5c47c8c9b-vlqwk_b5cca3ab-2749-4c33-b0d7-ed2fe69bbd92/copy-manifests/0.log" Sep 10 14:56:28.238200 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:28.238170 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv_131c2791-30c9-480d-9ac2-19739506c833/istio-proxy/0.log" Sep 10 14:56:28.482396 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:28.482363 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-2ctcp_c25c4402-a1d2-4635-8806-a2a598642bfe/kube-auth-proxy/0.log" Sep 10 14:56:30.196828 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:30.196785 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-92vmv_131c2791-30c9-480d-9ac2-19739506c833/istio-proxy/0.log" Sep 10 14:56:30.429922 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:30.429894 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-79c7cbc6bf-2ctcp_c25c4402-a1d2-4635-8806-a2a598642bfe/kube-auth-proxy/0.log" Sep 10 14:56:32.530360 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:32.530333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m7qjh_c0fb0680-1595-43fa-ac3d-c338139b3eb4/migrator/0.log" Sep 10 14:56:32.546746 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:32.546705 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m7qjh_c0fb0680-1595-43fa-ac3d-c338139b3eb4/graceful-termination/0.log" Sep 10 14:56:34.100987 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.100957 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/kube-multus-additional-cni-plugins/0.log" Sep 10 14:56:34.119249 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.119225 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/egress-router-binary-copy/0.log" Sep 10 14:56:34.136295 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.136270 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/cni-plugins/0.log" Sep 10 14:56:34.152590 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.152571 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/bond-cni-plugin/0.log" Sep 10 14:56:34.169676 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.169658 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/routeoverride-cni/0.log" Sep 10 14:56:34.186437 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.186419 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/whereabouts-cni-bincopy/0.log" Sep 10 14:56:34.202567 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.202546 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pv52q_048e8329-979a-4219-8b27-c2f7a41ed82d/whereabouts-cni/0.log" Sep 10 14:56:34.368278 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.368206 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hmvj6_563c20c6-a7e1-412b-8682-95009c41620c/kube-multus/0.log" Sep 10 14:56:34.420966 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.420938 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q996v_16991ff6-9847-4586-b642-2db22ae33842/network-metrics-daemon/0.log" Sep 10 14:56:34.434124 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:34.434098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q996v_16991ff6-9847-4586-b642-2db22ae33842/kube-rbac-proxy/0.log" Sep 10 14:56:35.195993 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.195962 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-controller/0.log" Sep 10 14:56:35.208139 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.208113 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/0.log" Sep 10 14:56:35.208255 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.208160 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovn-acl-logging/1.log" Sep 10 14:56:35.221623 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.221600 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/kube-rbac-proxy-node/0.log" Sep 10 14:56:35.237588 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.237568 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/kube-rbac-proxy-ovn-metrics/0.log" Sep 10 14:56:35.253495 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.253466 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/northd/0.log" Sep 10 14:56:35.269317 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.269300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/nbdb/0.log" Sep 10 14:56:35.287558 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.287541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/sbdb/0.log" Sep 10 14:56:35.335585 ip-10-0-135-236 kubenswrapper[2576]: I0910 14:56:35.335562 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrjvz_7e1410e4-456b-4092-afc8-e850a944a16a/ovnkube-controller/0.log"